:W&Bá¾A‹óM w6podt1tª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0·ewƒ‚" 1wgr1uwz21i5 w6podt1tŠß w6podt1tescheuller-uc-san-diego spatop-sweep"ž _wandb‚{}  batch_size‚2048  dropout‚0.27266866430165615  hidden_dim‚128 ' learning_rate‚0.0005820227119208908  num_embedding_layers‚6Bdpfx_vanilla_w6podt1tbnqupkqclj%dp-spatop-sweepagent-vanilla-v2-h9cgsŠ àíìÒØž£š3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64mÅ; w6podt1t*  _wandb‚ {"runtime":0}  _runtime‚0ïžúbÒÜ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 àíìÒÐò°Ü*-of*>/data/spatop/options_files/sweep_fixed_dp/config_w6podt1t.json*-l* /scratch/logs*-n*dpfx_vanilla_w6podt1t2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº p6yd0phr29vlzt6gz05arhdvej6c610iôzM¿ÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€€ì𳪀àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º p6yd0phr29vlzt6gz05arhdvej6c610iµª½Ç2  requirements.txt¾ÉíW w6podt1tZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64²ƒ$ Y w6podt1tZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64€7G*  w6podt1tb trainer/global_step2Úu+Ý\  w6podt1tZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64IÏÂ/  w6podt1tb*"trainer/global_step2¥ÑA K  w6podt1tj: áíìÒ°’*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] j=ד  w6podt1tj  áíìÒ˜ÒËÔî â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 865 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 19.0 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ êCÔj\ w6podt1tjÊ áíìÒ˜½ÏÕ·Trainable params: 21.2 M Non-trainable params: 50 Total params: 21.2 M Total estimated model params size (MB): 84.814 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 œÚ÷ÿq w6podt1tj` áíìÒø´ñÕPwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). •¥0û{ w6podt1tjj áíìÒ°¨ýÕZwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). vã\q w6podt1tj` áíìÒØÖ…ÖPwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). Q±Gàt w6podt1tjc áíìÒ°û‹ÖSwandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ì¼àn w6podt1tj] áíìÒ ˜’ÖMwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). £éð w6podt1t*Þ  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"Ö~ Në w6podt1tjÙ áíìÒÈû¦ÚÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ³¸­N w6podt1tj¼ áíìÒÈÈ·Ú«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ¡q\¶ë w6podt1tjÙ äíìÒ莃”È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. Ž¾U­P w6podt1tj¾ äíìÒðÛ ”­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ¯Òà“‚(:‰ ðíìÒÂÌÇï gpu.0.gpu‚84.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤È’‚(:ˆ ðíìÒÏÞ‹÷# proc.memory.rssMB‚ 2637.91015625 disk.nvme0n1.in‚ 64.390625 disk.rbd1.in‚0 disk.sda.out‚0.25 disk.sda1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 715.2109375) disk./.usagePercent‚39.52622103919889 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚26.06910334771647 network.sent‚33939 disk.drbd2165.out‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚0.25 disk.dm-0.out‚ 5.56396484375 disk.rbd1.out‚0.078125 disk.nvme0c0n1.in‚ 64.390625* proc.memory.percent‚16.100525856018066 disk.dm-1.in‚0 disk.dm-0.in‚ 64.390625 disk.sda2.in‚0$ disk.nvme0c0n1.out‚ 5.56396484375 network.recv‚42267$ memory_percent‚95.63469886779785" disk.nvme0n1.out‚ 5.56396484375$ disk./.usageGB‚588.9692573547363 disk.dm-1.out‚0.25 disk.sda2.out‚0ÁïÞ†ž‚(:” ÿíìÒÂüÄí gpu.0.gpu‚74.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.461799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡n¡3‘‚(:‡ ÿíìÒµùï disk.nvme0n1.out‚ 9.2890625 disk.sda1.out‚0 network.recv‚45627 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚0.65625* proc.memory.percent‚16.100668907165527* disk./.usagePercent‚39.526220783191356 disk.dm-1.in‚0$ memory_percent‚95.63412666320801 network.sent‚42387 disk.nvme0c0n1.in‚ 64.390625 disk.nvme0c0n1.out‚ 9.2890625 disk.sda1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 64.390625 disk.sda3.in‚0 cpu‚25.966029675299595 disk.dm-0.out‚ 9.2890625 disk.rbd1.out‚0.078125' proc.memory.availableMB‚ 715.3046875$ disk./.usageGB‚588.9692535400391 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2637.93359375 disk.dm-1.out‚0.65625 disk.drbd2165.in‚0 disk.dm-0.in‚ 64.390625 disk.sda3.out‚0.65625²W'“‚(:‰ ŽîìÒÀÁîî gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.8608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž÷l»š‚(: ŽîìÒ•‡´ï disk.rbd1.out‚0.140625 disk.nvme0c0n1.out‚ 15.234375 disk.dm-0.in‚ 64.390625 network.recv‚48891 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 0.95703125# proc.memory.rssMB‚ 2637.94921875( proc.memory.availableMB‚ 714.40234375 disk.drbd2165.in‚0 disk.nvme0n1.out‚ 15.234375 disk.sda.out‚ 0.95703125$ memory_percent‚95.63963413238525 disk.dm-1.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 0.95703125 network.sent‚46959 disk.nvme0n1.in‚ 64.390625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 64.390625 disk.sda1.in‚0 cpu‚25.905679554073284 disk.dm-0.out‚ 15.234375) disk./.usagePercent‚39.52623102349273$ disk./.usageGB‚588.9694061279297 disk.sda2.in‚0* proc.memory.percent‚16.100764274597168ˆCy3b/ lr-AdamW/pg1"trainer/global_step2:JPnýQß3 b/ lr-AdamW/pg2"trainer/global_step2:JP‚ãgž-!) _wandbruntime‚45  _runtime‚45–À"» % _timestamp‚1.7843627674405813e+09 _runtime‚ 45.947894789 _step‚0 & lr-AdamW/pg1‚7.129778221030912e-05 & lr-AdamW/pg2‚7.129778221030912e-05 trainer/global_step‚49:;½þÂ#½ _step‚0 & lr-AdamW/pg1‚7.129778221030912e-05 & lr-AdamW/pg2‚7.129778221030912e-05 trainer/global_step‚49 % _timestamp‚1.7843627674405813e+09 _runtime‚ 45.947894789 V$x$bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP0Ârs%bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JP¡€£bi&be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPOÕ²=h'bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP{Dr7d(b` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP^O['s)bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPÎ9+Àm*bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP°”„åi+be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JPšµª€f,bb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP?ÛÉ=e-ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP4Žp.bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP¸F¬­m/bi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP‹'µ0j0bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPŠÌs1bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPܳ%Èe2ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP¦«ônq3bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP2kÇQk4bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPЖÁñj5bf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPí ½Éa6b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPsͤCk7bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPß‘^'g8bc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP›´´"c9b_ bd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPj@`#n?bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPpš‚ll@bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPXÐheAba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPãj7ÓhBbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JP’ûR£fCbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPÔÆ=inDbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPGãýiEbe BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPävugFbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPçÝÁvGbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP°ÎÔŠlHbh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPV^™×rIbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP[íbenJbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP=ÎfÈjKbf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPi×ñnLbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPd­d£nMbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP/ åÄpNbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPˆ„¤eOba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPý™oákPbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPWÿHjQbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP®œÉ{bRb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP"@p²hSbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPiû·égTbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP§WárkUbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPbR¬eVba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP–PAJlWbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPè§ómXbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPü#ÝnYbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPþ0N gZbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPbew™p[bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP÷4¢l\bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP¥6-F_]b[ 8DeviceStatsMonitor.on_train_batch_start/num_device_alloc"trainer/global_step2:JPtØXi^be BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP{¿Ç×p_bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPó×U¿n`bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPUJ*Á^abZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP’y„ãgbbc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPuª¬³hcbd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP¥³þ5rdbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP. |mebi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP¿ A_fb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JPNË!vcgb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1847985796608 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385460 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7360817410048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚461154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚451528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 trainer/global_step‚49 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7449520251904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702768640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74076672 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7412365087078 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385462 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚108898 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554577408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784362767441444e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚202026 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 _runtime‚ 45.948046094 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7449162053120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437ž°‰'¡ƒO BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11748 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 51627787264 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 1847985796608 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚385460 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7360817410048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚461154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚451528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554577408 trainer/global_step‚49 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7449520251904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702768640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚74076672 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7412365087078 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚385462 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚108898 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7449162053120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554577408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784362767441444e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚202026 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 _runtime‚ 45.948046094 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7449162053120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 GDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPÈÁvØo¬bj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPY°õÒe­b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPù:º»d®b_ ]Ëb²b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP*ÖJ^³bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP®ÐYei´bd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPóÏnBnµbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPzZˆf¶ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPª.e·b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP”-KUm¸bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPeÔú=m¹bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPÇŸÁlºbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JPö‘ùf»ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP½SJÇp¼bk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JPœ ^½bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPtDÐg¾bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPR~j¿be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP8ÙhÀbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPl‚`ljÁbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPØ$ôQhÂbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPºV"iÃbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JP)›djcÄb^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JP ˆ÷mÅbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP¨¨oÆbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPæ‚Þ©rÇbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPŒ¼`iÈbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPy¿ÊkÉbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPå#1bÊb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JP·f°kËbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP·ºýÂmÌbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP̾“ËuÍbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP'óŒmÎbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP©R»eÏb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP`©mÐbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPš~ ÞgÑbb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP­ PmÒbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPÚÍ™]iÓbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPHhkmÔbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPÌ3-fÕba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP ÎzBlÖbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPï4Îl×bg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPæ{ûJwØbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP·£EDfÙba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JPÃúPlÚbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPz =uaÛb\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPK}QmÜbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPö`vkÝbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPÔ¼³lÞbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JP£ZÃÄ]ßbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP£–Ãiàbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JP«œ|fába >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JP0UÌ1mâbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP¤Gn iãbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JP |lØ`äb[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPý§qoåbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JP Zd²jæbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP$¨hçbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPZ¦Âkèbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP™]gdéb_ DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPso%Êoëbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP`ë'rìbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP; ’kíbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP³’ð{jîbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JPV-7×gïbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPÔÅ=nðbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP<´ÿhñbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JP éÕlòbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JP ýæXwóbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPgù5 sôbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP¿¾ªÖeõb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JP¤ú†ænöbi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPÔe› m÷bh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP¹sÚª_øbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPÞQ6qùbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP–ŽËrúbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÔ $gûbb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPö­i>düb_ bb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPÜÿ£jbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPªÝc‘b^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPwÃ’f’ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP‡ê¯¡e“b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPÍrír”bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPiÐ,Be•b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPh‹d–b_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7545133494272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚460580 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 34)os+ 1166592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚393094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚96795136 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393094 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚110935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 $ _timestamp‚1.784362768077103e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7559529224950 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚470220 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚448 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1882441566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1828845561856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚460580 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚206065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7559886812784 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7597773915648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 52299254784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 53697074688 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 53614616576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268Ì}â–”&ŸŽM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚460580 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚206065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 7559886812784 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 52640421376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7507328081920 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7597773915648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 52299254784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 53697074688 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 53614616576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7545116454912 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚470220 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 52640421376 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7597415709696 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚393092 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 7545133494272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7545133494272 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚460580 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚393094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚96795136 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚393094 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7545116454912 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚853672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚110935 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82458112 $ _timestamp‚1.784362768077103e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7559529224950 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚470220 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚205808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚448 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1882441566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1828845561856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0sÔXQA b< loss/FRt1/assignment_loss"trainer/global_step2:JP´ÕO‰A¡b< loss/FRt2/assignment_loss"trainer/global_step2:JPïÛC¢b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP{ 1A£b< loss/FBt1/assignment_loss"trainer/global_step2:JPùiA5B¤b= loss/SRqqt1/detection_loss"trainer/global_step2:JPõNT±@¥b; loss/FBt1/detection_loss"trainer/global_step2:JP€‹öpC¦b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP¶šÆ@§b; loss/FRt1/detection_loss"trainer/global_step2:JP´£>Ï@¨b; loss/FRt2/detection_loss"trainer/global_step2:JP†|MB©b= loss/SRqqt2/detection_loss"trainer/global_step2:JP÷w¬{@ªb; loss/FBt2/detection_loss"trainer/global_step2:JPžúî7«b2 loss/total_loss"trainer/global_step2:JP âÈßA¬b< loss/FBt2/assignment_loss"trainer/global_step2:JPu,¯‰-­b( epoch"trainer/global_step2:JPŽ|cw.®) _wandbruntime‚46  _runtime‚46XÍQ…ø¯ò 1loss/SRqqt1/assignment_loss‚4.227107903940214 /loss/FRt1/detection_loss‚0.3870065982130621 /loss/FRt2/detection_loss‚1.5325966440537508 /loss/SRqqt2/detection_loss‚1.35564021523041 /loss/FBt2/detection_loss‚10.768642618238342 %loss/total_loss‚68.59058733310266 0loss/FBt2/assignment_loss‚1.7337243136812428 epoch‚0 trainer/global_step‚49 _step‚3 /loss/FRt1/assignment_loss‚4.491249209076819 /loss/FRt2/assignment_loss‚5.346844649695336 1loss/SRqqt2/assignment_loss‚4.212587358129759 0loss/FBt1/assignment_loss‚1.6853315573112637 _runtime‚ 46.584719755 1loss/SRqqt1/detection_loss‚4.6970575979823685 /loss/FBt1/detection_loss‚28.152798667550087 $ _timestamp‚1.784362768078893e+09‚xCü°ö 1loss/SRqqt1/assignment_loss‚4.227107903940214 /loss/FRt1/detection_loss‚0.3870065982130621 /loss/FRt2/detection_loss‚1.5325966440537508 /loss/SRqqt2/detection_loss‚1.35564021523041 /loss/FBt2/detection_loss‚10.768642618238342 %loss/total_loss‚68.59058733310266 0loss/FBt2/assignment_loss‚1.7337243136812428 epoch‚0 trainer/global_step‚49 _step‚3 /loss/FRt1/assignment_loss‚4.491249209076819 /loss/FRt2/assignment_loss‚5.346844649695336 1loss/SRqqt2/assignment_loss‚4.212587358129759 0loss/FBt1/assignment_loss‚1.6853315573112637 _runtime‚ 46.584719755 1loss/SRqqt1/detection_loss‚4.6970575979823685 /loss/FBt1/detection_loss‚28.152798667550087 $ _timestamp‚1.784362768078893e+09&¡Ÿ±‚(:” îìÒÅä”í gpu.0.gpu‚75.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.309' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.461799999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bh±Ì©²‚(:ž îìÒø€‘ð disk.drbd2165.in‚0 disk.dm-0.in‚ 64.390625 disk.nvme0c0n1.in‚ 64.390625 disk.dm-1.out‚ 1.10546875 network.sent‚58478$ disk./.usageGB‚588.9694595336914 disk.nvme0n1.in‚ 64.390625( proc.memory.availableMB‚ 713.47265625 disk.dm-0.out‚19.15771484375 disk.sda.out‚ 1.10546875 disk.sda3.in‚0$ memory_percent‚95.64530849456787 disk.rbd0.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚54829# disk.nvme0n1.out‚19.15771484375 disk.sda2.in‚0 disk.rbd1.out‚0.140625 disk.sda1.out‚0 disk.dm-1.in‚0* disk./.usagePercent‚39.526234607598205 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚19.15771484375 disk.sda3.out‚ 1.10546875 disk.rbd0.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2638.00390625) proc.memory.percent‚16.10109806060791 cpu‚25.86883675172273ñbŠB”³‚(:‰ ¬îìÒÓñöì gpu.0.gpu‚43.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#¡ ´‚(:• ¬îìÒž¢»ï disk.nvme0n1.in‚ 132.2890625 cpu‚25.84130660829784 disk.dm-0.in‚ 132.2890625 disk.sda3.out‚1.484375 proc.cpu.threads‚17 disk.dm-0.out‚ 24.107421875 disk.rbd1.out‚0.21875# proc.memory.rssMB‚ 2638.01171875 disk.rbd0.in‚0! disk.nvme0n1.out‚ 24.107421875! disk.nvme0c0n1.in‚ 132.2890625 disk.sda1.out‚0) proc.memory.percent‚16.10114574432373 network.sent‚105940) disk./.usagePercent‚39.52621796710848 disk.sda2.out‚0 disk.sda.in‚0 network.recv‚59745 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚95.64676284790039& proc.memory.availableMB‚ 713.234375 disk.dm-1.out‚1.484375 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 24.107421875 disk.sda.out‚1.484375 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚588.9692115783691î)«¹.µ) _wandbruntime‚87  _runtime‚87L>Pö½ % _timestamp‚1.7843628088700724e+09 _runtime‚ 87.376151355 _step‚4 ' lr-AdamW/pg1‚0.00014405062120042048 ' lr-AdamW/pg2‚0.00014405062120042048 trainer/global_step‚9983'Ç·Á ' lr-AdamW/pg1‚0.00014405062120042048 ' lr-AdamW/pg2‚0.00014405062120042048 trainer/global_step‚99 % _timestamp‚1.7843628088700724e+09 _runtime‚ 87.376151355 _step‚4öûå.¸) _wandbruntime‚87  _runtime‚87|Þ³¢'¹œO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14788515167586 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14759926906880 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚394378 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚394121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚905980 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚767062 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚213743 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚99 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14788157579864 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3466836117504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 87.377438505 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14759926906880 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3570969017856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3466854729728 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 104025531392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14861909036544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚915606 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 102323289088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚213997 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14759943946240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1673040 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 101982129664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚767062 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102323289088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 104114288128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚767060 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 102164163938 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101823615576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚915606 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14861909036544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚14759926906880 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚180381 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 3570969017856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 3466854729728 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 104025531392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚14861909036544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚915606 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 102323289088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 101982129664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚213997 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14759943946240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1673040 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 101982129664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚767062 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102323289088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 104114288128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14686351003648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚767060 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 102164163938 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14862267235328 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 101823615576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14759943946240 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚180378 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚915606 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚767060 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚394121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0š<¼.») _wandbruntime‚88  _runtime‚88G.÷r«&¼¥M QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102834669298 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1699366 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 103335923200 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚924672 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3605318577664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 103175217772 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15010520899072 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 105053701632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚774694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774692 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚99 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102994756608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3500264876032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14907167936512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15010162693120 _step‚6 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774694 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 103335923200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚215802 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640A¥s ¯&½©M LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚915032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1699366 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 103335923200 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚924672 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3605318577664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15010520899072 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 103175217772 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚397925 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 102994756608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15010520899072 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 105053701632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚774694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚774692 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚99 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 102994756608 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3500264876032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14907167936512 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3605423842304 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14832844636160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15010162693120 _step‚6 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚774694 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 103335923200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚215802 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 105140354048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14935679305458 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102834669298 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268%¿Fü.¾) _wandbruntime‚88  _runtime‚88X›_ýú¿ô /loss/FRt2/detection_loss‚1.4171624269706524 epoch‚0 _runtime‚ 88.006926678 .loss/FRt1/assignment_loss‚4.57092024286588 /loss/FRt2/assignment_loss‚5.359864594289009 1loss/SRqqt1/assignment_loss‚2.722025571198299 1loss/FBt1/assignment_loss‚0.46545422659503255 0loss/SRqqt1/detection_loss‚4.198977889644047 2loss/SRqqt2/assignment_loss‚2.5671563744544983 /loss/FBt1/detection_loss‚29.548916841758057 .loss/FBt2/detection_loss‚4.256167949548847 trainer/global_step‚99 1loss/FBt2/assignment_loss‚0.25222658423226085 /loss/FRt1/detection_loss‚0.4286826093075231 0loss/SRqqt2/detection_loss‚1.114434773126141 %loss/total_loss‚56.90199008399025 % _timestamp‚1.7843628095011592e+09 _step‚7ÓÙëýþÀø %loss/total_loss‚56.90199008399025 % _timestamp‚1.7843628095011592e+09 _step‚7 /loss/FRt2/detection_loss‚1.4171624269706524 epoch‚0 _runtime‚ 88.006926678 .loss/FRt1/assignment_loss‚4.57092024286588 /loss/FRt2/assignment_loss‚5.359864594289009 1loss/SRqqt1/assignment_loss‚2.722025571198299 1loss/FBt1/assignment_loss‚0.46545422659503255 0loss/SRqqt1/detection_loss‚4.198977889644047 2loss/SRqqt2/assignment_loss‚2.5671563744544983 /loss/FBt1/detection_loss‚29.548916841758057 .loss/FBt2/detection_loss‚4.256167949548847 trainer/global_step‚99 1loss/FBt2/assignment_loss‚0.25222658423226085 /loss/FRt1/detection_loss‚0.4286826093075231 0loss/SRqqt2/detection_loss‚1.114434773126141ƒÁóÒ“Á‚(:ˆ »îìÒ‡í gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚282.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.499 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯Fq;«Â‚(:  »îìÒž“èð disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚0.21875 network.sent‚111423 disk.dm-0.in‚ 132.2890625 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.10119342803955 network.recv‚63626 disk.nvme0n1.in‚ 132.2890625# proc.memory.rssMB‚ 2638.01953125 cpu‚25.82003025083787) disk./.usagePercent‚39.52622283125163 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 1.8203125 disk.drbd2165.in‚0( proc.memory.availableMB‚ 712.55859375! disk.nvme0c0n1.in‚ 132.2890625% disk.nvme0c0n1.out‚27.19091796875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 1.8203125$ memory_percent‚95.65088748931885 proc.cpu.threads‚17$ disk./.usageGB‚588.9692840576172# disk.nvme0n1.out‚27.19091796875 disk.dm-0.out‚27.19091796875 disk.sda.out‚ 1.8203125 disk.rbd0.out‚0"°_”Â(:‰ ÊîìÒõ©Þí gpu.0.gpu‚62.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I Ƶ©Ä‚(:ž ÊîìÒ¬žªï disk.drbd2165.out‚0# disk.nvme0n1.out‚30.81201171875) proc.memory.percent‚16.10121726989746* disk./.usagePercent‚39.526235119613276 disk.rbd1.out‚0.34375! disk.nvme0c0n1.in‚ 132.2890625 disk.sda.out‚ 2.0546875 disk.drbd2165.in‚0 disk.dm-0.in‚ 132.2890625" proc.memory.rssMB‚ 2638.0234375$ disk./.usageGB‚588.9694671630859 disk.sda2.in‚0 disk.sda3.out‚ 2.0546875 disk.dm-1.out‚ 2.0546875 disk.rbd1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 712.578125 cpu‚25.804842253733256 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.in‚0# memory_percent‚95.6507682800293 network.sent‚158911 disk.sda2.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 132.2890625 disk.dm-0.out‚30.81201171875 network.recv‚68072% disk.nvme0c0n1.out‚30.81201171875 disk.sda.in‚04暎ŸÅ‚(:” ÙîìÒä÷›í gpu.0.gpu‚55.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚258.298' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.659600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QoÉ6Æ‚(: ÙîìÒ›îðî disk.sda.out‚2.359375 network.recv‚72206$ disk./.usageGB‚588.9692993164062 disk.nvme0n1.in‚ 132.2890625 disk.rbd1.out‚0.34375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.52622385528176 dis(éÙ†ek.sda2.out‚0! disk.nvme0c0n1.out‚ 33.6328125) proc.memory.percent‚16.10124111175537 cpu‚25.79242124872336 disk.rbd0.in‚0 disk.dm-0.out‚ 33.6328125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.out‚ 33.6328125 disk.sda.in‚0 disk.sda3.out‚2.359375 network.sent‚164344 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 132.2890625 disk.sda3.in‚0' proc.memory.availableMB‚ 712.4296875 disk.dm-1.out‚2.359375 disk.dm-0.in‚ 132.2890625 disk.sda1.out‚0$ memory_percent‚95.65167427062988# proc.memory.rssMB‚ 2638.02734375Ä<ƒ0Ç+ _wandbruntime‚128  _runtime‚128‰À·5ÅÈ¿ _runtime‚ 128.727023569 _step‚8 ' lr-AdamW/pg1‚0.00021680346019053182 ' lr-AdamW/pg2‚0.00021680346019053182 trainer/global_step‚149 % _timestamp‚1.7843628502209544e+09á+¢ÉÉà _step‚8 ' lr-AdamW/pg1‚0.00021680346019053182 ' lr-AdamW/pg2‚0.00021680346019053182 trainer/global_step‚149 % _timestamp‚1.7843628502209544e+09 _runtime‚ 128.727023569ÞKEè0Ê+ _wandbruntime‚128  _runtime‚128mŬ'˦O TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1148660 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1370058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 152780650846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 152677631488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1360432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 155550765056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 trainer/global_step‚149 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22274656019968 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 153018790912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5138292656128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5293733955072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22121978388480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22121978388480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1370058 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 153018790912 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚267628 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚267631 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90853888 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22275014218752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚319155 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22164665248094 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 _runtime‚ 128.728396871 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22121995427840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 % _timestamp‚1.7843628502222025e+09 _step‚9 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148662 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22275014218752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5138274043904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22011867557888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1360432 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 152677631488 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5293843421184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22011884597248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088*®Ï °'̪O QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22121995427840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 % _timestamp‚1.7843628502222025e+09 _step‚9 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1148662 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚22275014218752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5138274043904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚22011867557888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1148662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 103301120 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1360432 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 152677631488 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚586783 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1148660 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5293843421184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚22011884597248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚587054 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1148660 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1370058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109466112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 152780650846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 152677631488 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1360432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 155550765056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 152440102484 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 trainer/global_step‚149 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22274656019968 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚319423 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 153018790912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5138292656128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 5293733955072 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚22121978388480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22121978388480 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1370058 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22274656019968 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 153018790912 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚267628 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚267631 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90853888 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22275014218752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚22121995427840 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚319155 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚22164665248094 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 _runtime‚ 128.728396871 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 j;0Í+ _wandbruntime‚129  _runtime‚129Ô®!·&αM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 153451156206 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚149 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5171702802432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22269236457472 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 156570076672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1379124 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1369484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 129.360374982 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1369484 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22269219418112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22423267882496 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 153690258432 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚269376 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22422909676544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1156294 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1156292 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚321215 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1156292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5171721414656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 101070336 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 153791704680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154031425024 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¦‰'Â$ϵM XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 154031425024 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22158395269120 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 153451156206 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚149 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22269236457472 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5171702802432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22269236457472 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 156570076672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1379124 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1369484 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 129.360374982 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1369484 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22269219418112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22423267882496 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 153690258432 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 153690258432 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚269376 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚22312186973800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22422909676544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1156294 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22269219418112 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1379124 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1156292 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚321215 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1156292 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚590588 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚590864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5171721414656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 101070336 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5328190420992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 153791704680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 e]F0Ð+ _wandbruntime‚129  _runtime‚129® LüÑö _step‚11 /loss/FRt1/assignment_loss‚4.608585218847887 /loss/FRt2/assignment_loss‚5.189708725903631 1loss/SRqqt1/assignment_loss‚2.392111011579925 2loss/SRqqt2/assignment_loss‚2.1668262878389264 0loss/FBt2/assignment_loss‚0.1699532611964969 epoch‚0 _runtime‚ 129.361768483 .loss/FBt1/assignment_loss‚0.38589089780523 /loss/FRt1/detection_loss‚0.4980070569510535 trainer/global_step‚149 .loss/FRt2/detection_loss‚1.483145453180733 0loss/SRqqt1/detection_loss‚2.531537045235687 /loss/FBt1/detection_loss‚20.905981514095252 % _timestamp‚1.7843628508560758e+09 1loss/SRqqt2/detection_loss‚1.4671745482880083 /loss/FBt2/detection_loss‚3.9395260989504033 &loss/total_loss‚45.738447119873236°U–§Òú &loss/total_loss‚45.738447119873236 _step‚11 /loss/FRt1/assignment_loss‚4.608585218847887 /loss/FRt2/assignment_loss‚5.189708725903631 1loss/SRqqt1/assignment_loss‚2.392111011579925 2loss/SRqqt2/assignment_loss‚2.1668262878389264 0loss/FBt2/assignment_loss‚0.1699532611964969 epoch‚0 _runtime‚ 129.361768483 .loss/FBt1/assignment_loss‚0.38589089780523 /loss/FRt1/detection_loss‚0.4980070569510535 trainer/global_step‚149 .loss/FRt2/detection_loss‚1.483145453180733 0loss/SRqqt1/detection_loss‚2.531537045235687 /loss/FBt1/detection_loss‚20.905981514095252 % _timestamp‚1.7843628508560758e+09 1loss/SRqqt2/detection_loss‚1.4671745482880083 /loss/FBt2/detection_loss‚3.9395260989504033 ©ÅÓŸÓ‚(:” èîìÒÏðþî gpu.0.gpu‚86.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0voÎô¦Ô‚(:› èîìÒÊúìï disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0) proc.memory.percent‚16.10128879547119) disk./.usagePercent‚39.52623588763588 disk.sda2.out‚0 disk.rbd0.in‚0# memory_percent‚95.6552267074585 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 191.8359375( proc.memory.availableMB‚ 711.84765625$ disk.nvme0c0n1.out‚ 39.4052734375 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚75972 disk.rbd1.in‚0 disk.rbd1.out‚0.421875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-0.out‚ 39.4052734375 network.sent‚169773$ disk./.usageGB‚588.9694786071777 disk.sda2.in‚0" disk.nvme0n1.out‚ 39.4052734375# proc.memory.rssMB‚ 2638.03515625! disk.nvme0c0n1.in‚ 191.8359375 disk.dm-1.out‚2.515625 disk.dm-0.in‚ 191.8359375 cpu‚25.783416786869708 proc.cpu.threads‚17 disk.sda.out‚2.515625 disk.sda3.out‚2.515625É1ï]ŸÕ‚(:” ÷îìÒÙ˜í gpu.0.gpu‚83.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚261.294' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.258799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤œC¬Ö‚(:¡ ÷îìÒɶ ñ disk.rbd1.out‚0.421875) disk./.usagePercent‚39.52624535991464 disk.rbd0.in‚0# disk.nvme0n1.out‚43.07177734375 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 2.86328125 disk.dm-0.in‚ 191.8359375 disk.dm-0.out‚43.07177734375 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 191.8359375 disk.drbd2165.out‚0# memory_percent‚95.6432580947876 proc.cpu.threads‚17$ disk./.usageGB‚588.9696197509766 disk.sda1.in‚0( proc.memory.percent‚16.1013126373291( proc.memory.availableMB‚ 713.80859375 network.sent‚217357 disk.sda3.in‚0 disk.sda3.out‚ 2.86328125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 191.8359375% disk.nvme0c0n1.out‚43.07177734375 disk.sda.in‚0 disk.sda.out‚ 2.86328125 disk.sda1.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2638.0390625 cpu‚25.77439693879681 network.recv‚80845 disk.sda2.in‚0TR©’ׂ(:‡ †ïìÒĦ©î gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚221.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ù Ø‚(:• †ïìÒ‘·³ð) disk./.usagePercent‚39.52624075177903 disk.sda1.in‚0 disk.dm-0.in‚ 191.8359375# proc.memory.rssMB‚ 2638.04296875 network.sent‚222907 disk.sda.in‚0$ disk./.usageGB‚588.9695510864258! disk.nvme0c0n1.in‚ 191.8359375% disk.nvme0c0n1.out‚46.96630859375 disk.rbd0.out‚0 disk.dm-0.out‚46.96630859375 disk.sda2.in‚0 cpu‚25.77007708830668 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 191.8359375 disk.sda.out‚3.125 disk.dm-1.out‚3.125 disk.drbd2165.out‚0# disk.nvme0n1.out‚46.96630859375 disk.rbd1.out‚0.484375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 713.87109375 disk.sda3.out‚3.125 disk.dm-1.in‚0$ memory_percent‚95.64287662506104 proc.cpu.threads‚17 network.recv‚84700 disk.sda3.in‚0) proc.memory.percent‚16.10133647918701÷aø0Ù+ _wandbruntime‚169  _runtime‚169Á¢a’ÅÚ¿ ' lr-AdamW/pg1‚0.00028955629918064315 ' lr-AdamW/pg2‚0.00028955629918064315 trainer/global_step‚199 $ _timestamp‚1.784362891348548e+09 _runtime‚ 169.854715347 _step‚12™0ò‘ÉÛà ' lr-AdamW/pg1‚0.00028955629918064315 ' lr-AdamW/pg2‚0.00028955629918064315 trainer/global_step‚199 $ _timestamp‚1.784362891348548e+09 _runtime‚ 169.854715347 _step‚12 GAcï0Ü+ _wandbruntime‚169  _runtime‚169  :­'ݧO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843628913498883e+09 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29484046909440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7016635296256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29540457740880 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29687761202176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _runtime‚ 169.859768024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1530262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1824510 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29484046909440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203714292736 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚778546 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 203397137754 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚778827 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1530260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6809711970304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1814884 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1530260 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1814884 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 206815956992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29337418190848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚423946 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 206904713728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1824510 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 203714292736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7016527927296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29687403003392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29540815328602 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚354878 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203056589392"˜-YpÞ«O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7016527927296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚29687403003392 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚29540815328602 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203373133312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88756736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚354878 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 203056589392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 % _timestamp‚1.7843628913498883e+09 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚29687403003392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29484046909440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚524 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7016635296256 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚29540457740880 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚29687761202176 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _runtime‚ 169.859768024 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1530262 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107368960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1824510 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29484046909440 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203714292736 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚778546 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 203397137754 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚778827 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1530260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_b4ÝU/A atch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 6809711970304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚1814884 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1530260 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1814884 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 206815956992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29337418190848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚423946 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 206904713728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1824510 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 203714292736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚459 –s 0ß+ _wandbruntime‚170  _runtime‚170ÆßC·&à±M ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29483928862720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚356623 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚425789 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29687979466474 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204067643114 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29631287939072 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚782684 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7051082203136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚356626 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537894 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1833576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7050974841344 _step‚14 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3371470 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204726926848 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚426058 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204408191588 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207922862080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3361828 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204385760256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29631287939072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6843159341056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843628919770463e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29631270899712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204385760256¹E/»&áµM MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1537894 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚29836014865920 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1537892 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚782412 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚524 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204385760256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚459 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚29631287939072 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107361792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 6843159341056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843628919770463e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29631270899712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204385760256 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29483928862720 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚356623 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚425789 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚29687979466474 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 204067643114 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1537892 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29631287939072 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚782684 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7051082203136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29483911823360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚356626 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1537894 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1833576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7050974841344 _step‚14 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3371470 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204726926848 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚426058 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 103301120 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204408191588 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 207922862080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3361828 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36846098391040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 % _timestamp‚1.7843629324827795e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 254068635136 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36846081351680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚17 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36846081351680 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1911860 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528161 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36662934745088 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 253673076300 trainer/global_step‚249 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚36846098391040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 % _timestamp‚1.7843629324827795e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 254068635136 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 254409794560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36846081351680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚17 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36846081351680 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1911860 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚528161 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37100149986816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36662934745088 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 253673076300 trainer/global_step‚249 H·&ò±M J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8773481135104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 255081262080 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258902479872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113653248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36993322381312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37064487134816 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚ 211.621532153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95041024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8514578655232 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4197880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 258997520896 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚37064129546982 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36993339420672 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8773594788352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36809462456320 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 255081262080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 254684130022 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1919492 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚443873 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37248403643392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚249 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚529944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37248761849344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 255422428672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8514597267456 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚974066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 254684130022 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1919492 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚443873 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚530190 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37248403643392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚249 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚529944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37248761849344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 255422428672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8514597267456 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 255024678496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚974066 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚973817 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8773481135104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 255081262080 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 258902479872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1919492 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2288028 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113653248 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36993322381312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚37064487134816 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚ 211.621532153 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚95041024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8514578655232 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4197880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚443876 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2žÝ%€0ô+ _wandbruntime‚211  _runtime‚211Ëš°õú /loss/FRt2/assignment_loss‚4.094662545432507 1loss/FBt1/assignment_loss‚0.16662936882523985 0loss/SRqqt1/detection_loss‚3.122586288779272 .loss/FBt1/detection_loss‚9.683384588148604 .loss/FBt2/detection_loss‚5.433355003387017 epoch‚0 trainer/global_step‚249 _runtime‚ 211.622742789 2loss/SRqqt1/assignment_loss‚1.9388195691060046 0loss/FRt1/detection_loss‚0.40239463715011037 &loss/total_loss‚33.807337769032955 % _timestamp‚1.7843629331170268e+09 1loss/SRqqt2/assignment_loss‚1.584368077430823 1loss/SRqqt2/detection_loss‚1.1851158574320797 /loss/FRt1/assignment_loss‚4.522712003681022 1loss/FBt2/assignment_loss‚0.38544178109383725 /loss/FRt2/detection_loss‚1.2878680485664424 _step‚19«¶Vöþ _step‚19 /loss/FRt2/assignment_loss‚4.094662545432507 1loss/FBt1/assignment_loss‚0.16662936882523985 0loss/SRqqt1/detection_loss‚3.122586288779272 .loss/FBt1/detection_loss‚9.683384588148604 .loss/FBt2/detection_loss‚5.433355003387017 epoch‚0 trainer/global_step‚249 _runtime‚ 211.622742789 2loss/SRqqt1/assignment_loss‚1.9388195691060046 0loss/FRt1/detection_loss‚0.40239463715011037 &loss/total_loss‚33.807337769032955 % _timestamp‚1.7843629331170268e+09 1loss/SRqqt2/assignment_loss‚1.584368077430823 1loss/SRqqt2/detection_loss‚1.1851158574320797 /loss/FRt1/assignment_loss‚4.522712003681022 1loss/FBt2/assignment_loss‚0.38544178109383725 /loss/FRt2/detection_loss‚1.2878680485664424¥˜¹—”÷‚(:‰ ÂïìÒ＂í gpu.0.gpu‚46.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÑ©ø‚(:ž ÂïìÒ“®žï! disk.nvme0c0n1.in‚ 256.5859375 disk.rbd0.out‚0% proc.memory.availableMB‚ 713.65625* proc.memory.percent‚16.101360321044922 cpu‚25.751491868027376 network.recv‚101742 disk.sda.out‚ 4.52734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 4.52734375 disk.drbd2165.in‚0 network.sent‚287171% disk.nvme0c0n1.out‚68.79345703125 disk.drbd2165.out‚0# disk.nvme0n1.out‚68.79345703125 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 4.52734375 disk.dm-0.in‚ 256.5859375 disk.dm-0.out‚68.79345703125) disk./.usagePercent‚39.52623691166601$ disk./.usageGB‚588.9694938659668 disk.sda2.out‚0 disk.rbd0.in‚0# memory_percent‚95.6441879272461 disk.rbd1.out‚0.625 disk.nvme0n1.in‚ 256.5859375! proc.memory.rssMB‚ 2638.046875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0R«Œæ‘ù‚(:† ÑïìÒ”¼×ì gpu.0.gpu‚77.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚206.05' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.21 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WwÒ.¥ú‚(:š ÑïìÒÃñèï disk.sda3.out‚5.03125 disk.rbd0.out‚0 network.recv‚106782% disk.nvme0c0n1.out‚71.95166015625 disk.sda.out‚5.03125* disk./.usagePercent‚39.526239215733824 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 256.5859375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 256.5859375 disk.dm-0.out‚71.95166015625 network.sent‚334659 disk.sda3.in‚0$ memory_percent‚95.64425945281982( proc.memory.availableMB‚ 713.64453125! proc.memory.rssMB‚ 2638.046875 cpu‚25.746573249207707$ disk./.usageGB‚588.9695281982422 disk.rbd1.in‚0 disk.dm-1.out‚5.03125 proc.cpu.threads‚17 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 256.5859375 disk.sda2.out‚0 disk.rbd1.out‚0.625# disk.nvme0n1.out‚71.95166015625 disk.sda1.out‚0* proc.memory.percent‚16.101360321044922EϨ0û+ _wandbruntime‚252  _runtime‚252²pÃü½ _runtime‚ 252.113815361 _step‚20 & lr-AdamW/pg1‚0.0004350619771608659 & lr-AdamW/pg2‚0.0004350619771608659 trainer/global_step‚299 $ _timestamp‚1.784362973606939e+09ý…ŒÇýÁ & lr-AdamW/pg1‚0.0004350619771608659 & lr-AdamW/pg2‚0.0004350619771608659 trainer/global_step‚299 $ _timestamp‚1.784362973606939e+09 _runtime‚ 252.113815361 _step‚20°e#¿0þ+ _wandbruntime‚252  _runtime‚252™¿{³'ÿ­O WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43988485378048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44513255169024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10461630603264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2293462 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305105296384 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2293460 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2293460 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 304764136960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44208132833280 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1161638 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10461746360832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 309139925504 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 304289563208 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44292757901896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10152606435328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10152587823104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44512896970240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44208149872640 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚632525 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2733414 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2723788 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44293115489618 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 252.113895562 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 304764136960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚21 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2293462 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2723788 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44208149872640 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1161906 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚632525 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2733414 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2723788 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44293115489618 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 252.113895562 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 304764136960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚21 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2293462 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305105296384 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43988468338688 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2723788 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44513255169024 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44512896970240 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341159424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10461630603264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2293462 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305105296384 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2293460 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2293460 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 304764136960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚44208132833280 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1161638 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10461746360832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚632260 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 309139925504 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 304289563208 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44292757901896 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚44208149872640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10152606435328 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10152587823104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528˜À&0+ _wandbruntime‚252  _runtime‚252ØaÁùü‚·M E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44355390902272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843629742388377e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1165695 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 305776763904 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚531123 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚22 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 310159502848 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 310070753280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44134996049920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚634290 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10496087334912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2732840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚531126 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =aŒ|üÁ7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44440637215324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44355373862912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10186035193856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2301094 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 306117930496 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305300616930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528Ö%²1Á&ƒ»M NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1165695 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 305776763904 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚531123 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _step‚22 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 310159502848 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2301094 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10186016581632 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 310070753280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚44134996049920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚44355373862912 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚634290 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10496087334912 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2732840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚531126 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44355390902272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305641165404 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1165413 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44440637215324 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚44355373862912 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10186035193856 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2301094 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚299 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 306117930496 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 305300616930 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚44355390902272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843629742388377e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736(àÊ0„+ _wandbruntime‚252  _runtime‚252ÍRmþ…ø 0loss/FBt2/assignment_loss‚0.3550007789459154 1loss/SRqqt1/detection_loss‚3.4934425590271947 1loss/SRqqt2/detection_loss‚0.9359739738851489 trainer/global_step‚299 /loss/FRt2/assignment_loss‚4.040225999183904 2loss/SRqqt1/assignment_loss‚1.7234610214077046 /loss/FRt2/detection_loss‚1.2884233244804337 .loss/FBt2/detection_loss‚21.83994853411402 /loss/FRt1/assignment_loss‚4.045901652599903 1loss/SRqqt2/assignment_loss‚1.510884300248661 1loss/FBt1/assignment_loss‚0.27508925237584947 /loss/FRt1/detection_loss‚0.4603269944408457 .loss/FBt1/detection_loss‚2.093021401145445 % _timestamp‚1.7843629742401652e+09 _runtime‚ 252.745850147 _step‚23 %loss/total_loss‚42.06169979185503 epoch‚0Øf´†ü _step‚23 %loss/total_loss‚42.06169979185503 epoch‚0 0loss/FBt2/assignment_loss‚0.3550007789459154 1loss/SRqqt1/detection_loss‚3.4934425590271947 1loss/SRqqt2/detection_loss‚0.9359739738851489 trainer/global_step‚299 /loss/FRt2/assignment_loss‚4.040225999183904 2loss/SRqqt1/assignment_loss‚1.7234610214077046 /loss/FRt2/detection_loss‚1.2884233244804337 .loss/FBt2/detection_loss‚21.83994853411402 /loss/FRt1/assignment_loss‚4.045901652599903 1loss/SRqqt2/assignment_loss‚1.510884300248661 1loss/FBt1/assignment_loss‚0.27508925237584947 /loss/FRt1/detection_loss‚0.4603269944408457 .loss/FBt1/detection_loss‚2.093021401145445 % _timestamp‚1.7843629742401652e+09 _runtime‚ 252.745850147—ü²Ò”‡‚(:‰ àïìÒÇ°¢í gpu.0.gpu‚86.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.249' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.8498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…'&ÿ§ˆ‚(:œ àïìÒßàêî disk.sda3.out‚5.28125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 317.1015625 disk.sda1.out‚0 network.sent‚340789 disk.sda2.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚76.55810546875# proc.memory.rssMB‚ 2638.05078125 disk.dm-0.in‚ 317.1015625! disk.nvme0c0n1.in‚ 317.1015625% disk.nvme0c0n1.out‚76.55810546875* proc.memory.percent‚16.101384162902832 cpu‚25.744261391411282$ disk./.usageGB‚588.9697113037109 disk.sda2.out‚0 disk.sda.out‚5.28125$ memory_percent‚95.64485549926758& proc.memory.availableMB‚ 713.546875 disk.dm-1.out‚5.28125 disk.rbd0.out‚0 disk.dm-0.out‚76.55810546875 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚110860 disk.dm-1.in‚0 disk.rbd1.out‚0.765625) disk./.usagePercent‚39.52625150409547 disk.sda3.in‚0 disk.sda.in‚0él¢Ÿ‰‚(:” ïïìÒææ÷î gpu.0.gpu‚67.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gne­Š‚(:¢ ïïìÒø£üð disk.nvme0n1.in‚ 317.1015625" disk.nvme0n1.out‚ 80.1650390625 disk.sda1.in‚0 disk.dm-1.out‚ 5.59765625 disk.sda2.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 713.15234375 cpu‚25.74077176536765 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2638.05078125* proc.memory.percent‚16.101384162902832$ disk./.usageGB‚588.9699554443359 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 5.59765625 disk.dm-1.in‚0 network.sent‚389718 disk.dm-0.in‚ 317.1015625 disk.dm-0.out‚ 80.1650390625# memory_percent‚95.6472635269165 network.recv‚115629) disk./.usagePercent‚39.52626788857766 disk.rbd1.out‚0.765625! disk.nvme0c0n1.in‚ 317.1015625$ disk.nvme0c0n1.out‚ 80.1650390625 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 5.59765625 proc.cpu.threads‚17âVÑ‚(:‰ þïìÒÃ×çî gpu.0.gpu‚46.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s¸w«Œ‚(:  þïìÒµõ¦ñ disk.rbd1.in‚0 disk.dm-1.out‚ 5.9453125 disk.drbd2165.out‚0 disk.rbd1.out‚0.828125 disk.sda1.in‚0# disk./.usageGB‚588.969669342041 disk.dm-0.in‚ 317.1015625$ disk.nvme0c0n1.out‚ 84.4833984375 disk.sda.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 713.30859375* proc.memory.percent‚16.101431846618652 proc.cpu.threads‚17 network.recv‚119154 disk.sda3.in‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 317.1015625# proc.memory.rssMB‚ 2638.05859375# memory_percent‚95.6463098526001* disk./.usagePercent‚39.526248688012586 disk.dm-0.out‚ 84.4833984375 disk.sda.out‚ 5.9453125 disk.sda1.out‚0 disk.sda3.out‚ 5.9453125 disk.dm-1.in‚0" disk.nvme0n1.out‚ 84.4833984375 cpu‚25.739546025571826 network.sent‚395130 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 317.1015625Î~ÿC0+ _wandbruntime‚293  _runtime‚293 †’»ÄŽ¾ trainer/global_step‚349 % _timestamp‚1.7843630146510208e+09 _runtime‚ 293.157006242 _step‚24 & lr-AdamW/pg1‚0.0005078148161509773 & lr-AdamW/pg2‚0.0005078148161509773$ øœÈ _step‚24 & lr-AdamW/pg1‚0.0005078148161509773 & lr-AdamW/pg2‚0.0005078148161509773 trainer/global_step‚349 % _timestamp‚1.7843630146510208e+09 _runtime‚ 293.157006242Á ˆ0+ _wandbruntime‚293  _runtime‚293Ö]’¾³'‘­O BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51926002152448 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 360356646400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2675060 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51926002152448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51570184314880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2675062 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 trainer/global_step‚349 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51925643953664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚25 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51314001932288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51570184314880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0|r’±O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3187866 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 % _timestamp‚1.7843630146521137e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 355800798208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 G?DeviceStatsMonitá7þ±E$or.on_train_batch_start/segment.small_pool.peak‚268 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3187866 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚737077 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51925643953664 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3178240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 360356646400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2675060 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358198784 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚51926002152448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11824025749504 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 355246598478 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51570184314880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2675062 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚737342 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 trainer/global_step‚349 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2675060 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚51925643953664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51570201354240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12184293639168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚25 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51314001932288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51570184314880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51926002152448 HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115750400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51717425344512 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _runtime‚ 293.785269127 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚739171 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51717425344512 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3196932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11857454508032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51460529643520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 356813432320 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52074255816192 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2682694 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 355917103838 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3187292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51816429707998 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12218849119232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51816787295832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187292 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52073897610240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51717442383872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51717442383872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 356813432320 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1357822 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 trainer/global_step‚3496!³·Á&•»M KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 356472265728 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 356813432320 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1357822 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 104712704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 trainer/global_step‚349 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115750400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51717425344512 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 _runtime‚ 293.785269127 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341166592 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚739171 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51717425344512 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3196932 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11857454508032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51460529643520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 356813432320 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702769664 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52074255816192 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5869984 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2682694 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2682692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 355917103838 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3187292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1357544 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51816429707998 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12218849119232 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51816787295832 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9640 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3187292 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52073897610240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51717442383872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51717442383872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437Þ{$0–+ _wandbruntime‚293  _runtime‚293‘¥p—ú 1loss/SRqqt2/assignment_loss‚1.442475885810221 1loss/FBt1/assignment_loss‚0.26254470172571875 /loss/FRt1/detection_loss‚0.4418256469322319 /loss/FRt2/detection_loss‚1.3160789137981066 epoch‚0 _runtime‚ 293.786799746 /loss/FRt1/assignment_loss‚4.108915274193946 1loss/FBt2/assignment_loss‚0.11158909535173807 .loss/FBt1/detection_loss‚5.108281811104234 % _timestamp‚1.7843630152810957e+09 /loss/FRt2/assignment_loss‚3.955950897672902 2loss/SRqqt1/assignment_loss‚1.7500173263929106 .loss/FBt2/detection_loss‚7.558806551827325 trainer/global_step‚349 _step‚27 1loss/SRqqt1/detection_loss‚3.2590028137307274 1loss/SRqqt2/detection_loss‚0.8331924437731205 &loss/total_loss‚30.148681362313184j=$˜þ /loss/FRt2/assignment_loss‚3.955950897672902 2loss/SRqqt1/assignment_loss‚1.7500173263929106 .loss/FBt2/detection_loss‚7.558806551827325 trainer/global_step‚349 _step‚27 1loss/SRqqt1/detection_loss‚3.2590028137307274 1loss/SRqqt2/detection_loss‚0.8331924437731205 &loss/total_loss‚30.148681362313184 1loss/SRqqt2/assignment_loss‚1.442475885810221 1loss/FBt1/assignment_loss‚0.26254470172571875 /loss/FRt1/detection_loss‚0.4418256469322319 /loss/FRt2/detection_loss‚1.3160789137981066 epoch‚0 _runtime‚ 293.786799746 /loss/FRt1/assignment_loss‚4.108915274193946 1loss/FBt2/assignment_loss‚0.11158909535173807 .loss/FBt1/detection_loss‚5.108281811104234 % _timestamp‚1.7843630152810957e+09a°y”™‚(:‰ ðìÒî gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘(L®š‚(:£ ðìÒÔÁäï disk.nvme0n1.in‚ 317.1015625 disk.sda2.out‚0 network.sent‚398490) disk./.usagePercent‚39.52624254383177# proc.memory.rssMB‚ 2638.05859375* proc.memory.percent‚16.101431846618652 disk.sda3.out‚ 6.2421875# disk.nvme0n1.out‚87.74560546875 disk.dm-0.out‚87.74560546875 disk.rbd1.out‚0.828125! disk.nvme0c0n1.in‚ 317.1015625 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 317.1015625 disk.sda2.in‚0' proc.memory.availableMB‚ 713.3046875% disk.nvme0c0n1.out‚87.74560546875 disk.sda1.in‚0 disk.dm-1.out‚ 6.2421875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.out‚0 cpu‚25.737200742384413 network.recv‚122750 disk.rbd1.in‚0 disk.sda.out‚ 6.2421875 disk.drbd2165.in‚0 disk.rbd0.in‚0$ memory_percent‚95.64633369445801$ disk./.usageGB‚588.9695777893066 disk.sda1.out‚0 disk.dm-1.in‚0ªÙ©ž›‚(:“ œðìÒ‚”ëî gpu.0.gpu‚86.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y4Ô¬œ‚(:’ œðìÒÉ­‚ñ disk.sda1.in‚0( proc.memory.availableMB‚ 713.71484375$ disk.nvme0c0n1.out‚ 92.8232421875* proc.memory.percent‚16.101455688476562$ \ ñdisk./.usageGB‚588.9697647094727 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 384.78125) disk./.usagePercent‚39.52625508820095 disk.sda.out‚6.453125 disk.sda1.out‚0 disk.dm-1.out‚6.453125 proc.memory.rssMB‚ 2638.0625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda3.out‚6.453125 disk.dm-0.in‚ 384.78125$ memory_percent‚95.64383029937744 disk.sda2.in‚0 network.sent‚447924 disk.nvme0c0n1.in‚ 384.78125 disk.drbd2165.in‚0 disk.dm-0.out‚ 92.8232421875 cpu‚25.73462851583779 disk.rbd1.in‚0 disk.rbd1.out‚0.90625 network.recv‚127426 disk.rbd0.out‚0" disk.nvme0n1.out‚ 92.8232421875 proc.cpu.threads‚17Xèžð”‚(:‰ «ðìÒ«Þ’ï gpu.0.gpu‚54.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ^¾§ž‚(:œ «ðìÒæÃñ disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 384.78125 disk.sda2.in‚0% disk.nvme0c0n1.out‚96.39794921875# proc.memory.rssMB‚ 2638.06640625 disk.rbd0.in‚0 disk.nvme0n1.in‚ 384.78125 disk.rbd1.out‚0.90625 disk.sda1.in‚0# memory_percent‚95.6470251083374 disk.nvme0c0n1.in‚ 384.78125 disk.sda3.in‚0 disk.dm-1.out‚ 6.7578125 disk.dm-0.out‚96.39794921875 disk.sda1.out‚0( proc.memory.availableMB‚ 713.19140625 disk.sda2.out‚0 disk.rbd1.in‚0 network.recv‚131617 disk.sda3.out‚ 6.7578125# disk.nvme0n1.out‚96.39794921875 disk.sda.in‚0 disk.sda.out‚ 6.7578125* proc.memory.percent‚16.101479530334473$ disk./.usageGB‚588.9695091247559 network.sent‚451284 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚25.733087318770675 proc.cpu.threads‚17) disk./.usagePercent‚39.52623793569615ä~W0Ÿ+ _wandbruntime‚334  _runtime‚334„€S:Ä ¾ & lr-AdamW/pg2‚0.0005805676551410886 trainer/global_step‚399 % _timestamp‚1.7843630556891575e+09 _runtime‚ 334.195102616 _step‚28 & lr-AdamW/pg1‚0.0005805676551410886´È¡Â trainer/global_step‚399 % _timestamp‚1.7843630556891575e+09 _runtime‚ 334.195102616 _step‚28 & lr-AdamW/pg1‚0.0005805676551410886 & lr-AdamW/pg2‚0.0005805676551410886Áò8Þ0¢+ _wandbruntime‚334  _runtime‚334íòÅ^³'£­O JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3632692 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58639552565248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59338749135872 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58639535525888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843630556901786e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚399 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3056662 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3642318 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58932235796480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3056662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 406155140608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59338390937088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚29 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1545670 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚703881 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 405522537024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚841789 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Œ¸·'¤±O ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13906997859840 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13906880005120 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13495463675904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358198784 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58932252835840 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58932235796480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9626 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117854720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341159424 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6698980 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13495482288128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 406496300032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 411515571712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3056660 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59338390937088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59338390937088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚29 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58932252835840 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1545670 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚703881 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 405522537024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚841789 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 411416329216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3632692 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702769664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340548362 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1545385 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11750 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 104712704 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553966455 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554578432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357587722 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536055063 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58639552565248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59338749135872 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59045415650634 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59045058062912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚841507 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58639535525888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843630556901786e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚399 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11750 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3056662 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3642318 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554578432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58932235796480 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3056662 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9626 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702769664 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 406155140608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 412535432192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 407508934144 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13941347505664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59192579788506 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1549482 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59079493865472 _step‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 $ _timestamp‚1.784363056320205e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚843856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58786063237120 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59079476826112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 trainer/global_step‚399 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3651384 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59487002799616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 407167767552 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13528911046656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59192937376340 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 407167767552 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3641744 J@DeviceStatsMonitor.on_train_batch_end/alloc(³Oºation.small_pool.peak‚11750 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚843610 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169Xè:º¿&§¹M QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98973184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3641744 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 412535432192 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 407508934144 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553966455 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13941347505664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3064292 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59192579788506 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340548474 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1549482 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚705626 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚59079493865472 _step‚30 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358205952 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536055063 $ _timestamp‚1.784363056320205e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1549233 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚843856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357587834 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58786063237120 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6706036 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3064294 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59079476826112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 trainer/global_step‚399 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3651384 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚59487002799616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3064294 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3651384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702769664 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 407167767552 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554578432 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554578432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚463 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341166592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358205952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 407508934144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13528911046656 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59192937376340 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 407167767552 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3641744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11750 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚843610 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚59079476826112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚59487002799616 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚59486644593664 F*ƒû©õ _runtime‚ 334.827424732 _step‚31 /loss/FRt1/assignment_loss‚4.062368127679697 1loss/SRqqt2/assignment_loss‚1.511189905672884 /loss/FRt1/detection_loss‚0.4126345918481512 /loss/SRqqt1/detection_loss‚2.89197317321108 trainer/global_step‚399 0loss/FRt2/assignment_loss‚3.9249515559362327 /loss/FBt1/detection_loss‚2.3503243643535723 .loss/FBt2/detection_loss‚6.096002564248112 %loss/total_loss‚25.94029159223059 epoch‚0 2loss/SRqqt1/assignment_loss‚1.7971425875327232 0loss/FBt1/assignment_loss‚0.1679747473444031 0loss/FBt2/assignment_loss‚0.7196938014886756 .loss/FRt2/detection_loss‚1.291838485556913 1loss/SRqqt2/detection_loss‚0.7141976873581466 $ _timestamp‚1.784363056321428e+09Úá¼—ÿªù $ _timestamp‚1.784363056321428e+09 _runtime‚ 334.827424732 _step‚31 /loss/FRt1/assignment_loss‚4.062368127679697 1loss/SRqqt2/assignment_loss‚1.511189905672884 /loss/FRt1/detection_loss‚0.4126345918481512 /loss/SRqqt1/detection_loss‚2.89197317321108 trainer/global_step‚399 0loss/FRt2/assignment_loss‚3.9249515559362327 /loss/FBt1/detection_loss‚2.3503243643535723 .loss/FBt2/detection_loss‚6.096002564248112 %loss/total_loss‚25.94029159223059 epoch‚0 2loss/SRqqt1/assignment_loss‚1.7971425875327232 0loss/FBt1/assignment_loss‚0.1679747473444031 0loss/FBt2/assignment_loss‚0.7196938014886756 .loss/FRt2/detection_loss‚1.291838485556913 1loss/SRqqt2/detection_loss‚0.7141976873581466[Á[“«‚(:ˆ ºðìÒäøÊï gpu.0.gpu‚25.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÑW±§¬‚(:œ ºðìÒ£ƒò disk.dm-0.in‚ 384.78125 disk.sda.in‚0 cpu‚25.837814428313784 proc.cpu.threads‚17 network.sent‚456580 disk.dm-1.out‚ 7.0546875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 384.78125 disk.rbd1.out‚0.96875 network.recv‚135223 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.out‚0$ memory_percent‚95.66795825958252( proc.memory.availableMB‚ 709.76171875) disk./.usagePercent‚39.52626532850231 disk.sda.out‚ 7.0546875$ disk./.usageGB‚588.9699172973633 disk.sda1.in‚0$ disk.nvme0n1.out‚103.57275390625! disk.dm-0.out‚103.57275390625 proc.memory.rssMB‚ 2640.8125* proc.memory.percent‚16.118240356445312 disk.rbd0.out‚0 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 384.78125 disk.rbd1.in‚0& disk.nvme0c0n1.out‚103.57275390625 disk.sda3.in‚0 disk.sda3.out‚ 7.0546875 disk.drbd2165.in‚0 disk.drbd2165.out‚0ѱ‹uN­bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP‹ØÀÂH®bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPŠjN¯bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPì‚Ö@°b; particle/accuracy_2_of_5"trainer/global_step2:JP‡×àú@±b; particle/accuracy_4_of_5"trainer/global_step2:JPFÄž]H²bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP%´±°H³bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPªQH´bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPu¯)žJµbE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPxJ{@¶b; particle/accuracy_3_of_3"trainer/global_step2:JPŸÙ‚J·bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPdk“J¸bE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPfŸ5sH¹bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPw°þ\NºbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPyUïŠH»bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP˺7ïH¼bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPIj#dH½bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÔïN¾bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP¸ƒèH¿bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPÀOHÀbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÞ›×NÁbI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP`m9«JÂbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPçXÛ}JÃbE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPX¸yzHÄbC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPþžÝ4NÅbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP÷½$ýHÆbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP}|ÍHJÇbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŸtÐ+HÈbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPtKmJÉbE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPö͇JÊbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¶Dð:NËbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPj'AÏ@Ìb; particle/accuracy_4_of_4"trainer/global_step2:JPã±JÍbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPý£^ NÎbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP˜°¼HÏbC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÖ…¶;Ðb6 jet/accuracy_1_of_4"trainer/global_step2:JP+J%?JÑbE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP5·ÀJJÒbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPdáSHÓbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÖ‹HÔbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPlâHÕbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÕŸù@Öb; particle/accuracy_1_of_2"trainer/global_step2:JPÓäsN×bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP¯ÇèˆNØbI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP"HÙbC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPä§;Úb6 jet/accuracy_2_of_2"trainer/global_step2:JP´Ð@Ûb; particle/accuracy_2_of_4"trainer/global_step2:JPž=ðJÜbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP?ƒuNÝbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP¬žÀHÞbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP†ÎoPJßbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPc…[2JàbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP ÆWsJábE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPð ÏJâbE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÝ¢Ï`HãbC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPsììNäbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPt\ñ°JåbE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÜ‹5.;æb6 jet/accuracy_2_of_4"trainer/global_step2:JPø»µ;çb6 jet/accuracy_4_of_4"trainer/global_step2:JP¸ögJèbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPìyJébE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¢SÍKJêbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¦ ùJëbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP:;:JìbE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP_¾-JíbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPƒÒw$JîbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP$/qJïbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPž©çÉHðbC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPN•öNñbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPÂì°5NòbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP€bvJóbE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP¤LÃNôbI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP]À¥y;õb6 jet/accuracy_2_of_3"trainer/global_step2:JPÅî"BJöbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP¡U?1N÷bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP*}÷^HøbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP&çnHùbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPg¤T^JúbE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP*žfNûbI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP³j}%HübC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP­dJýbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPþ;h^NþbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP^ðJÿbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPþÓ—ŠN€bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP N,HNbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPD…ˆúJ‚bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¶÷JƒbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPžß%ZH„bC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPŸüúŽH…bC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPײÂbN†bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP ‡p;N‡bI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPįÛpJˆbE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP”<0•J‰bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP´ºªJŠbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP =ŽÓH‹bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPëi; NŒbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPð¬¸HbC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPçºJŽbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPáë‰VHbC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP\<ÝšHbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPòª÷oN‘bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP\'ênN’bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPYƒ¸J“bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPl‘íJ”bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPHx„ÖJ•bE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPæÒ H–bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPl¹úJ—bE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP®g?ÍJ˜bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP}´sJ™bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP¸¸×µNšbI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPGÂØÕN›bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¯W+ÀJœbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÌÍJbE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP„q—_HžbC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP8F[JŸbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPgfTH bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPq¥«QJ¡bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPM¤ëžH¢bC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP·vz¼J£bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP;êoJ¤bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPÛ]£N¥bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPMàBúH¦bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP¿-J§bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP4moÚH¨bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPŒT(N©bI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP=ŽŸJªbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPïøº¹J«bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP-~ÑbJ¬bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPšÊŵH­bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP RH®bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP×=NJ¯bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŽµÐüJ°bE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPeß`SJ±bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÓr)J²bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP_þwH³bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPQgFH´bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPŸkûJµbE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPõ¹J¶bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‰;H·bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP‘ìŽ.J¸bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPHKêJ¹bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÀ%‹NºbI &Purity/1FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP4$–ÞJ»bE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPª(¦H¼bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPˆv¾;½b6 jet/accuracy_3_of_4"trainer/global_step2:JPH$þ¼J¾bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPý÷|N¿bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP²C?‹;Àb6 jet/accuracy_1_of_1"trainer/global_step2:JPðÛÄHÁbC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP^9T`NÂbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP4`þÿJÃbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPx,æmJÄbE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPç\ÍHÅbC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPWœúNÆbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPfÉí;Çb6 jet/accuracy_1_of_3"trainer/global_step2:JPÂ:‰7JÈbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPH¢øHÉbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP†ÐCHÊbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¹‡HËbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPäa!JÌbE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPP11MHÍbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP†Ì}@Îb; particle/accuracy_2_of_2"trainer/global_step2:JPá Å«NÏbI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPƒ¥9JÐbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPwžJÑbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPƋׄJÒbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP !†ÃNÓbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP\aŒHÔbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP¥Šÿ HÕbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPfè ;Öb6 loss/val_total_loss"trainer/global_step2:JP&¬±iJ×bE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP<ØÔNØbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPz–¯@Ùb; particle/accuracy_3_of_5"trainer/global_step2:JPøÇŽˆNÚbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPlf•ÏNÛbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPQ¥¹/HÜbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPcè[çJÝbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPí§{JÞbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPê‘„HßbC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP#JKHàbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP äHábC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP¾ä¦¤@âb; particle/accuracy_1_of_1"trainer/global_step2:JPÞŒŒß@ãb; particle/accuracy_3_of_4"trainer/global_step2:JPðúmÔ;äb6 jet/accuracy_2_of_5"trainer/global_step2:JPx¹W8NåbI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP:d¶JæbE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPm†RNçbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP»Y/Š;èb6 jet/accuracy_3_of_3"trainer/global_step2:JP+¢g;éb6 jet/accuracy_5_of_5"trainer/global_step2:JP•‰eÍJêbE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP䶼QHëbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÌ„}æ@ìb; particle/accuracy_1_of_5"trainer/global_step2:JPâ!ËRHíbC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPŒ]OCHîbC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP%•#cNïbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPOÏÓHðbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÆ°ƒHñbC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÇÆŠgJòbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPiµSHóbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPËú*¶<ôb7 particle/sensitivity"trainer/global_step2:JP æ‰É8õb3 particle/f_score"trainer/global_step2:JPB{¢uNöbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPühF3J÷bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPÊ~ˆÞNøbI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP4±ÝƒNùbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPC°/…<úb7 particle/specificity"trainer/global_step2:JP¥ÚºÔNûbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP*a<éNübI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPjf HýbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP8Pi NþbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP–¡‡ÂHÿbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP¿¤œH€bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP8[JbE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPìz¦éN‚bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPg'~HƒbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP-'ÐH„bC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP³Ò*²N…bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP?o!H†bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP>LÆõH‡bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPÏv_&HˆbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP4Žž N‰bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP…£‡JŠbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPMâÐ.N‹bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPWôÉJŒbE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP,ZNbI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPS¸JŽbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP/všJbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP^ ôNbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP·†ùJ‘bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPK±²J;’b6 jet/accuracy_4_of_5"trainer/global_step2:JPúòXgH“bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPœÿ—G”bB validation_average_jet_accuracy"trainer/global_step2:JPTp%N•bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP$Às4H–bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP=“—~J—bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPg)UÑN˜bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPåH™bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPû=‚JšbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¦ Ì$J›bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPN.ÖHœbC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPAk˜MNbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP§úsvNžbI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPyšJ@Ÿb; particle/accuracy_1_of_3"trainer/global_step2:JPý—Â@ b; particle/accuracy_2_of_3"trainer/global_step2:JP¶OiH¡bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP°¯ÇJ¢bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÃû¤9£bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_sõz6ktep2:JP‰–ÅN¤bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP’ï#9N¥bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPžÜŒ„9¦b4 particle/accuracy"trainer/global_step2:JPWh…@§b; particle/accuracy_5_of_5"trainer/global_step2:JP­J¨bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP`V‰N©bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÀžÍ–HªbC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÈÀAN«bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¿èJ¬bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPý-*J­bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¯§¨jJ®bE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPˆH¯bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPìyzOJ°bE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPúäáoN±bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPO<2…N²bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPæNÈJ³bE "Purity/*FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP¡Þ¨¸J´bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP› HµbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP ”Ì6H¶bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP[7ÙJ·bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP’ "J¸bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPr×öQJ¹bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPb*z­;ºb6 jet/accuracy_3_of_5"trainer/global_step2:JPBâ¼@J»bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPæcŠ\J¼bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPò·!éH½bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPb/üRH¾bC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP¾&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4206516146659851 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.635554850101471 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6789828538894653 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008530220948159695 jet/accuracy_4_of_5‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.10441095381975174 % _timestamp‚1.7843630742219257e+09 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.09061171114444733 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.03262428566813469 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.1271509826183319 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48148149251937866 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07425764948129654 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9850307703018188 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.00287438603118062 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9706070423126221 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.7697792053222656 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.1067361980676651 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.20000000298023224 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.34821027517318726 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10126553475856781 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16906683146953583 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0019568894058465958 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.19908350706100464 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2777777910232544 jet/accuracy_3_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7545378804206848 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.36326801776885986 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9844202399253845 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.0806673988699913 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1369614452123642 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 epoch‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.48148149251937866 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9279742240905762 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.20026688277721405 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9793680310249329 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.10225597769021988 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.04755114018917084 +jet/accuracy_1_of_2‚0.26959651708602905 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9887377619743347 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.16294409334659576 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚0 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.030160389840602875 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07463961839675903 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9919333457946777 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9990025758743286 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.031904079020023346 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.1597127765417099 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.05324074253439903 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.08087383210659027 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9890016913414001 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.09780842065811157 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1909349113702774 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.028678270056843758 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07085749506950378 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.11576490104198456 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.15752080082893372 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.009238663129508495 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9605512022972107 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.009659258648753166 *jet/accuracy_1_of_4‚0.5312197804450989 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9734505414962769 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.09852471947669983 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9435563683509827 particle/accuracy_1_of_2‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.08076804876327515 +jet/accuracy_2_of_2‚0.04286031052470207 particle/accuracy_2_of_4‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9761243462562561 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08306712657213211 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7541611194610596 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07574135065078735 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.1179317981004715 +jet/accuracy_2_of_4‚0.12112933397293091 jet/accuracy_4_of_4‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 trainer/global_step‚399 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.058570969849824905 ;"Purity/*FRt1SRqqt0FBt/event_purity‚0.056307822465896606 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8211632966995239 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9903704524040222 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.11183372884988785 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.09865614026784897 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.19075047969818115 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.011198350228369236 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_3‚0.11675290018320084 _step‚32 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.06879188865423203 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0783444195985794 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.04366021975874901 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2734375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0714285746216774 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.00629398925229907 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.14362192153930664 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0755208358168602 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21404501795768738 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3756084442138672 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.12550941109657288 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.5833333134651184 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.012172241695225239 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.007395012769848108 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9718078374862671 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1308899074792862 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1056768074631691 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8719735145568848 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9678642749786377 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4310453534126282 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2222222238779068 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.00957124587148428 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.125875785946846 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08842158317565918 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9784684777259827 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.00967324897646904 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.11441820114850998 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6111111044883728 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.06625000387430191 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5092592239379883 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.00501253129914403 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19075047969818115 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08339603990316391 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9359867572784424 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9620644450187683 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.009659258648753166 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.15472392737865448 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0625 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3891844153404236 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11405853182077408 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5092592239379883 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.012075550854206085 -jet/accuracy_3_of_4‚0.0029761905316263437 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 +jet/accuracy_1_of_1‚0.07946263253688812 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9461172223091125 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.006183101795613766 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_3‚0.5251860618591309 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5425786375999451 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.976233720779419 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.08090723305940628 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08101359009742737 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.042380936443805695 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.18887709081172943 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21298101544380188 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12431973218917847 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9731672406196594 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7115384340286255 *loss/val_total_loss‚26.504225275370167 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_3_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9905241131782532 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.09166989475488663 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0028272380586713552 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.15627363324165344 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9903704524040222 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9970238208770752 jet/accuracy_2_of_5‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6111111044883728 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,jet/accuracy_3_of_3‚0.007712919730693102 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.21199443936347961 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01211707666516304 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07873769104480743 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9229156970977783 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.03126969560980797 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0017449381994083524 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07085749506950378 +particle/sensitivity‚0.5177100300788879 (particle/f_score‚0.49132323265075684 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.09780842065811157 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *particle/specificity‚0.838715136051178 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07503006607294083 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1D„Xm"5ÏÖ{ :"Purity/0FRt2SRqqt0FBt/event_purity‚0.00629398925229907 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.14362192153930664 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.0755208358168602 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.21404501795768738 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3756084442138672 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.12550941109657288 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.5833333134651184 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.012172241695225239 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.007395012769848108 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9718078374862671 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.1308899074792862 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1056768074631691 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8719735145568848 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9678642749786377 ("Purity/2FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.4310453534126282 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9956710338592529 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.2222222238779068 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.00957124587148428 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.125875785946846 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.08842158317565918 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9784684777259827 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.00967324897646904 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.11441820114850998 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6111111044883728 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.06625000387430191 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5092592239379883 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.00501253129914403 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19075047969818115 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.08339603990316391 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9359867572784424 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9620644450187683 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.009659258648753166 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.15472392737865448 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.0625 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.3891844153404236 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.11405853182077408 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5092592239379883 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.012075550854206085 -jet/accuracy_3_of_4‚0.0029761905316263437 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 +jet/accuracy_1_of_1‚0.07946263253688812 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9461172223091125 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.006183101795613766 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_3‚0.5251860618591309 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5425786375999451 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.976233720779419 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.08090723305940628 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08101359009742737 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 9 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.042380936443805695 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.18887709081172943 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.21298101544380188 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.12431973218917847 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9731672406196594 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7115384340286255 *loss/val_total_loss‚26.504225275370167 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_3_of_5‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9905241131782532 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.09166989475488663 <"Purity/*FRt2SRqqt*FBt/event_purity‚0.0028272380586713552 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.15627363324165344 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9903704524040222 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9970238208770752 jet/accuracy_2_of_5‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6111111044883728 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,jet/accuracy_3_of_3‚0.007712919730693102 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.21199443936347961 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01211707666516304 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07873769104480743 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9229156970977783 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.03126969560980797 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0017449381994083524 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07085749506950378 +particle/sensitivity‚0.5177100300788879 (particle/f_score‚0.49132323265075684 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.09780842065811157 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *particle/specificity‚0.838715136051178 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07503006607294083 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.02845148555934429 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.6342592835426331 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.07948402315378189 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 _runtime‚ 352.728620299 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9637888073921204 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.12148165702819824 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9870940446853638 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4206516146659851 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.635554850101471 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6789828538894653 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.008530220948159695 jet/accuracy_4_of_5‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.10441095381975174 % _timestamp‚1.7843630742219257e+09 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.09061171114444733 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.03262428566813469 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.1271509826183319 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48148149251937866 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07425764948129654 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9850307703018188 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.00287438603118062 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9706070423126221 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.7697792053222656 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.1067361980676651 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.20000000298023224 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.34821027517318726 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.10126553475856781 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.16906683146953583 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0019568894058465958 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.19908350706100464 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2777777910232544 jet/accuracy_3_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7545378804206848 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.36326801776885986 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9844202399253845 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.0806673988699913 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1369614452123642 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 epoch‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.48148149251937866 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9279742240905762 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.20026688277721405 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9793680310249329 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.10225597769021988 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.04755114018917084 +jet/accuracy_1_of_2‚0.26959651708602905 jet/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9887377619743347 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.16294409334659576 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚0 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.030160389840602875 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07463961839675903 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9919333457946777 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9990025758743286 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt*SRqqt1FBt/event_purity‚0.031904079020023346 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.1597127765417099 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.05324074253439903 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.08087383210659027 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9890016913414001 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.09780842065811157 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.1909349113702774 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.028678270056843758 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07085749506950378 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.11576490104198456 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.15752080082893372 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.009238663129508495 ("Purity/2FRt*SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9605512022972107 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.009659258648753166 *jet/accuracy_1_of_4‚0.5312197804450989 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9734505414962769 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.098524719ùºôº47669983 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9435563683509827 particle/accuracy_1_of_2‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.08076804876327515 +jet/accuracy_2_of_2‚0.04286031052470207 particle/accuracy_2_of_4‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9761243462562561 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.08306712657213211 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7541611194610596 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07574135065078735 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.1179317981004715 +jet/accuracy_2_of_4‚0.12112933397293091 jet/accuracy_4_of_4‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 trainer/global_step‚399 ;"Purity/*FRt1SRqqt*FBt/event_purity‚0.058570969849824905 ;"Purity/*FRt1SRqqt0FBt/event_purity‚0.056307822465896606 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8211632966995239 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9903704524040222 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.11183372884988785 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.09865614026784897 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.19075047969818115 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.011198350228369236 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_3‚0.11675290018320084 _step‚32 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.06879188865423203 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0783444195985794 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.04366021975874901 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2734375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.0714285746216774 7ꬽžÐ‚(:“ ÉðìÒÐÝÄð gpu.0.gpu‚56.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉJìÑ‚(:’ ÉðìÒŠ¶Œõ* proc.memory.percent‚16.263318061828613 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17" disk.nvme0n1.out‚ 131.376953125# memory_percent‚96.9658374786377 disk.rbd0.in‚0 disk.dm-0.in‚ 384.78125 disk.dm-0.out‚ 131.376953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚7.40625 disk.sda1.out‚0 disk.rbd1.out‚0.96875 disk.dm-1.out‚7.40625 disk.sda3.in‚0 disk.dm-1.in‚0) disk./.usagePercent‚39.55842345890838$ disk./.usageGB‚589.4490966796875 disk.nvme0n1.in‚ 384.78125 disk.nvme0c0n1.in‚ 384.78125' proc.memory.availableMB‚ 497.1171875 network.sent‚504350 disk.sda3.out‚7.40625 cpu‚25.906727444580294 network.recv‚140206 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 131.376953125# proc.memory.rssMB‚ 2664.58203125¶R]”Ò‚(:‰ ØðìÒÂÕÖî gpu.0.gpu‚89.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚255.869' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cýiq¹Ó‚(:® ØðìÒŒ‡ôð network.sent‚561663 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 disk.rbd0.out‚0 cpu‚25.899541741592767* disk./.usagePercent‚39.610152341276965 disk.sda.out‚ 7.69921875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 428.40234375 disk.dm-0.in‚ 428.40234375$ memory_percent‚96.96600437164307# proc.memory.rssMB‚ 2664.58203125 network.recv‚143879 proc.cpu.threads‚17& disk.nvme0c0n1.out‚631.12646484375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚631.12646484375* proc.memory.percent‚16.263318061828613$ disk./.usageGB‚590.2198944091797" disk.nvme0c0n1.in‚ 428.40234375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.out‚1.046875 disk.sda3.out‚ 7.69921875 disk.dm-1.out‚ 7.69921875! disk.dm-0.out‚631.12646484375( proc.memory.availableMB‚ 497.08984375Á0[½”Ô‚(:‰ çðìÒè¥î gpu.0.gpu‚65.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôl·Õ‚(:¬ çðìÒ¬ƒÆï disk.sda.in‚0 disk.sda3.out‚ 8.30078125& disk.nvme0c0n1.out‚634.76318359375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 8.30078125! disk.dm-0.out‚634.76318359375( proc.memory.availableMB‚ 497.33203125) disk./.usagePercent‚39.60836182458249 disk.dm-0.in‚ 428.40234375 disk.sda2.in‚0$ memory_percent‚96.96452617645264$ disk.nvme0n1.out‚634.76318359375 disk.rbd1.out‚1.046875# proc.memory.rssMB‚ 2664.58203125 network.recv‚147442 disk.drbd2165.out‚0 disk.sda.out‚ 8.30078125* proc.memory.percent‚16.263318061828613$ disk./.usageGB‚590.1932144165039 disk.drbd2165.in‚0 network.sent‚566192 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 428.40234375 cpu‚25.89129335129332 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 428.40234375 disk.sda2.out‚0.!Ž‘0Ö+ _wandbruntime‚395  _runtime‚395‰ËjÀ׺ $ lr-AdamW/pg1‚0.00058201373644528 $ lr-AdamW/pg2‚0.00058201373644528 trainer/global_step‚449 % _timestamp‚1.7843631168008437e+09 _runtime‚ 395.306869442 _step‚33h9¤”Äؾ _runtime‚ 395.306869442 _step‚33 $ lr-AdamW/pg1‚0.00058201373644528 $ lr-AdamW/pg2‚0.00058201373644528 trainer/global_step‚449 % _timestamp‚1.7843631168008437e+09!þ@º©0Ù+ _wandbruntime‚395  _runtime‚395‡µŸÛ®'Ú¨O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67873749372928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68223870967808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1770032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558928 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4167152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68698796005888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68223870967808 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 474925038080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚793648 _runtime‚ 395.308200205 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4167152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 474199297883 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473858748001 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4157177 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15182832314368 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 480764750336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15663504554496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92510208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 474583699968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚976128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚793651 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 474583699968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68347591081569 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4157177 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480690852352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68347948670811 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚34 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3558926 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 ! _timestamp‚1.784363116802e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558926 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68698437628416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68223853928448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0”õê²'Û¬O TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1769776 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚68347948670811 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3558928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15182813702144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚34 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3558926 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67873732333568 ! _timestamp‚1.784363116802e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3558926 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68698437628416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚68223853928448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 474925038080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚67873749372928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚976381 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚68223870967808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1770032 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3558928 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4167152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68698437628416 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68698796005888 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚68223870967808 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 474925038080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚793648 _runtime‚ 395.308200205 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚68223853928448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4167152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15663597064704 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 474199297883 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473858748001 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4157177 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15182832314368 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 480764750336 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚7716103 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15663504554496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92510208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 474583699968 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚976128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚793651 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68698796005888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 474583699968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚68347591081569 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4157177 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 480690852352 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 % _timestamp‚1.7843631174325078e+09 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚795396 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 _runtime‚ 395.9387696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15216261072896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚86211584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481809247744 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15698070320640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3566560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1773612 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15216261072896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68846691284992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68020243005440 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚86211584 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 481809247744 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15698070320640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 475596326912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚35 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 % _timestamp‚1.7843631174325078e+09 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚795396 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 _runtime‚ 395.9387696 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0#á}Ý0ß+ _wandbruntime‚395  _runtime‚395F¸ÐÁàû /loss/FBt1/detection_loss‚1.4880424733056032 /loss/FRt1/assignment_loss‚3.994069939446989 2loss/SRqqt1/assignment_loss‚1.6279783897551297 0loss/FBt1/assignment_loss‚0.1646293729803174 0loss/SRqqt1/detection_loss‚2.249259870607666 epoch‚1 % _timestamp‚1.7843631174342477e+09 _runtime‚ 395.939937755 _step‚36 .loss/FRt2/assignment_loss‚3.58819350020753 1loss/FBt2/assignment_loss‚0.48728419053885674 /loss/FRt2/detection_loss‚1.2723274165683607 1loss/SRqqt2/detection_loss‚0.7104010205238498 /loss/FBt2/detection_loss‚3.3092891549070678 trainer/global_step‚449 &loss/total_loss‚20.774413369375687 2loss/SRqqt2/assignment_loss‚1.4306438648393367 0loss/FRt1/detection_loss‚0.45229417569497793'0†áÿ 1loss/FBt2/assignment_loss‚0.48728419053885674 /loss/FRt2/detection_loss‚1.2723274165683607 1loss/SRqqt2/detection_loss‚0.7104010205238498 /loss/FBt2/detection_loss‚3.3092891549070678 trainer/global_step‚449 &loss/total_loss‚20.774413369375687 2loss/SRqqt2/assignment_loss‚1.4306438648393367 0loss/FRt1/detection_loss‚0.45229417569497793 /loss/FBt1/detection_loss‚1.4880424733056032 /loss/FRt1/assignment_loss‚3.994069939446989 2loss/SRqqt1/assignment_loss‚1.6279783897551297 0loss/FBt1/assignment_loss‚0.1646293729803174 0loss/SRqqt1/detection_loss‚2.249259870607666 epoch‚1 % _timestamp‚1.7843631174342477e+09 _runtime‚ 395.939937755 _step‚36 .loss/FRt2/assignment_loss‚3.58819350020753$çÚìŸâ‚(:” öðìÒÙÐßï gpu.0.gpu‚95.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚165.842' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.168400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dîǼã‚(:± öðìÒˆüýñ cpu‚25.884891993208935 network.recv‚151291% disk.nvme0n1.out‚1426.26220703125 disk.sda.in‚0 disk.sda.out‚ 8.71484375 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚584651* disk./.usagePercent‚39.558434723239884" disk.dm-0.out‚1426.26220703125 disk.rbd1.out‚1.109375 proc.cpu.threads‚17$ disk./.usageGB‚589.4492645263672 disk.rbd1.in‚0 disk.sda1.in‚0$ memory_percent‚96.96547985076904 disk.dm-1.out‚ 8.71484375 disk.dm-0.in‚ 428.40234375' disk.nvme0c0n1.out‚1426.26220703125* proc.memory.percent‚16.263318061828613 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 428.40234375" disk.nvme0c0n1.in‚ 428.40234375 disk.rbd0.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 497.17578125# proc.memory.rssMB‚ 2664.58203125 disk.sda3.in‚0 disk.sda3.out‚ 8.71484375 disk.rbd0.out‚0ƒ[ Úžä‚(:“ …ñìÒ÷š©ï gpu.0.gpu‚49.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Möå´å‚(:© …ñìÒ¢ÃÊñ disk.sda2.in‚0# disk.nvme0n1.out‚1429.517578125 disk.dm-0.out‚1429.517578125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 497.12109375 network.sent‚636708 disk.sda2.out‚0 disk.rbd1.out‚1.109375 disk.rbd0.out‚0* proc.memory.percent‚16.263318061828613$ memory_percent‚96.96581363677979 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 428.40234375 network.recv‚156311$ disk./.usageGB‚589.4226875305176% disk.nvme0c0n1.out‚1429.517578125 disk.sda.out‚ 9.12109375 disk.sda3.out‚ 9.12109375# proc.memory.rssMB‚ 2664.58203125" disk.nvme0c0n1.in‚ 428.40234375 disk.dm-1.out‚ 9.12109375 disk.drbd2165.in‚0 cpu‚25.87870660038639) disk./.usagePercent‚39.55665111874883 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 428.402343759åÞ”æ‚(:‰ ”ñìÒÕ¥”í gpu.0.gpu‚45.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚244.269' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.8538 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#`B¬ç‚(:¡ ”ñìÒù˜óî' disk.nvme0c0n1.out‚1437.12744140625 disk.dm-1.in‚0% disk.nvme0n1.out‚1437.12744140625' proc.memory.availableMB‚ 497.0859375 network.sent‚640026 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 481.34375# proc.memory.rssMB‚ 2664.58203125 proc.cpu.threads‚17) disk./.usagePercent‚39.55665521486937 disk.sda2.in‚0$ memory_percent‚96.96602821350098 cpu‚25.87264697148933 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚1.25 disk.sda3.out‚ 9.30078125 disk.dm-1.out‚ 9.30078125 disk.nvme0n1.in‚ 481.34375 network.recv‚159855 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 481.34375* proc.memory.percent‚16.263318061828613$ disk./.usageGB‚589.4227485656738 disk.rbd1.in‚0 disk.sda.out‚ 9.30078125 disk.rbd0.out‚0" disk.dm-0.out‚1437.12744140625Þ¿ X0è+ _wandbruntime‚436  _runtime‚436s•&Äé¾ & lr-AdamW/pg1‚0.0005819860741687962 & lr-AdamW/pg2‚0.0005819860741687962 trainer/global_step‚499 % _timestamp‚1.7843631577683067e+09 _runtime‚ 436.274272301 _step‚37 ³qêÈê % _timestamp‚1.7843631577683067e+09 _runtime‚ 436.274272301 _step‚37 & lr-AdamW/pg1‚0.0005819860741687962 & lr-AdamW/pg2‚0.0005819860741687962 trainer/global_step‚499%q¢+ã0ë+ _wandbruntime‚436  _runtime‚436-\iµ'ì¯O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 532167326720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3940526 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 525279201792 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚17386332474368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8552155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚38 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75724098751319 trainer/global_step‚499 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚880898 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 524475234909 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75585922449408 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17386437567488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16854270240768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16854251628544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75199282966528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75585905410048 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚880901 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 525620539904 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76111542989312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75585905410048 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4611629 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3940528 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 532080845824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75585922449408 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1960659 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843631577693844e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 525620539904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1079761 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 532080845824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75585922449408 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3940528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1960659 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843631577693844e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 525620539904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1079761 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚38 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75724098751319 trainer/global_step‚499 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8562132 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86480896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚880898 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 524475234909 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚75585922449408 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1960938 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚17386437567488 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚463 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16854270240768 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16854251628544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75199282966528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75585905410048 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76111542989312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76111184611840&Á‚Çô0î+ _wandbruntime‚436  _runtime‚436ž¨¬¾&ï¸M QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4630670 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 526633174016 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75733146439680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76259796653056 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3948158 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75733163479040 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4630670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16887698999296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚39 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3948160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76259438268416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1082107 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17420894022144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75871620477025 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75345793638400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 526291828736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843631583956275e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1081866 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1964509 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75345776599040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76259796653056 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8568839 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 526633174016 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚75733146439680 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1964753 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚882646 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚76259796653056 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3948158 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚75733163479040 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4630670 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚16887698999296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚39 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3948160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76259438268416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1082107 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚17420894022144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚75871620477025 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75345793638400 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 526291828736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843631583956275e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1081866 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4630670 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83523931595264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82524816560128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5076056 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚968148 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83100248831827 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19109129822208 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82947973931008 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 576316041728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82947956891648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19109024729088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9398184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83523931595264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚42 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83524289972736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1183121 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 575091721817 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83523931595264 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18525689554944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82524816560128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5076056 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚968148 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83100248831827 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4322128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19109129822208 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82947973931008 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 576316041728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82947956891648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚468 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19109024729088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9398184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83523931595264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚42 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 trainer/global_step‚549 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83524289972736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1183121 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 575091721817 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H¦Œ0þ+ _wandbruntime‚478  _runtime‚478&¬¿&ÿ¹M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83247412968179 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1185454 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83672543636480 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329758 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18559118313472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19143584126464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2155350 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 576987330560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83095197921280 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 trainer/global_step‚549 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82671310192640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83247770557533 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0élå倽M H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18559118313472 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19143584126464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 584352338432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2155350 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 576987330560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚83095197921280 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 trainer/global_step‚549 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5085122 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 577328675840 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 576987330560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚533 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚82671327232000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83095214960640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2155096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82671310192640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83247770557533 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4329760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83247412968179 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1185454 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83672543636480 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4329760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4329758 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989+¬0Ó0+ _wandbruntime‚478  _runtime‚478ŸÔ´¤‚ý /loss/FBt2/detection_loss‚1.6470092635049864 trainer/global_step‚549 _runtime‚ 478.07775503 1loss/FBt2/assignment_loss‚0.07327961396569865 /loss/FRt2/detection_loss‚1.4585834652817353 1loss/SRqqt1/detection_loss‚2.0768366254633293 0loss/FRt1/assignment_loss‚3.5980211072864554 /loss/FRt1/detection_loss‚0.6283854332332518 /loss/FBt1/detection_loss‚1.1487253144914427 _step‚44 /loss/FRt2/assignment_loss‚2.786177740282617 &loss/total_loss‚17.306876919663114 % _timestamp‚1.7843631995720377e+09 epoch‚1 2loss/SRqqt1/assignment_loss‚1.7399178633932024 2loss/SRqqt2/assignment_loss‚1.4441421399725245 1loss/FBt1/assignment_loss‚0.08242746798113537 1loss/SRqqt2/detection_loss‚0.6233708848067364&1üƒ 0loss/FRt1/assignment_loss‚3.5980211072864554 /loss/FRt1/detection_loss‚0.6283854332332518 /loss/FBt1/detection_loss‚1.1487253144914427 _step‚44 /loss/FRt2/assignment_loss‚2.786177740282617 &loss/total_loss‚17.306876919663114 % _timestamp‚1.7843631995720377e+09 epoch‚1 2loss/SRqqt1/assignment_loss‚1.7399178633932024 2loss/SRqqt2/assignment_loss‚1.4441421399725245 1loss/FBt1/assignment_loss‚0.08242746798113537 1loss/SRqqt2/detection_loss‚0.6233708848067364 /loss/FBt2/detection_loss‚1.6470092635049864 trainer/global_step‚549 _runtime‚ 478.07775503 1loss/FBt2/assignment_loss‚0.07327961396569865 /loss/FRt2/detection_loss‚1.4585834652817353 1loss/SRqqt1/detection_loss‚2.0768366254633293,:Wi”„‚(:‰ ÁñìÒ­íáï gpu.0.gpu‚95.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šä¯…‚(:¤ ÁñìÒ¡‚žò disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.856404104196223 proc.cpu.threads‚17 network.recv‚172420" disk.dm-0.out‚1452.92138671875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2664.58203125 disk.dm-0.in‚ 481.34375 disk.sda2.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 481.34375* proc.memory.percent‚16.263318061828613 network.sent‚713732$ disk./.usageGB‚589.4227180480957 disk.sda1.out‚0 disk.rbd1.out‚1.3125' disk.nvme0c0n1.out‚1452.92138671875 disk.sda.in‚0 disk.sda3.out‚ 10.3828125 disk.drbd2165.in‚0% disk.nvme0n1.out‚1452.92138671875 disk.nvme0c0n1.in‚ 481.34375 disk.dm-1.out‚ 10.3828125 disk.drbd2165.out‚0* disk./.usagePercent‚39.556653166809106 disk.sda2.out‚0$ memory_percent‚96.96817398071289 disk.rbd0.out‚0& proc.memory.availableMB‚ 496.734375 disk.rbd1.in‚0 disk.sda.out‚ 10.3828125’öƒr”†‚(:‰ ÐñìÒÿ™¡í gpu.0.gpu‚76.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z|Â+±‡‚(:¦ ÐñìÒ¿ÅÄï network.sent‚776892 disk.dm-1.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2664.5859375 cpu‚25.851689682315346) disk./.usagePercent‚39.55664932669609 disk.drbd2165.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚589.4226608276367 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 10.6640625 network.recv‚177003 disk.rbd0.in‚0$ memory_percent‚96.97315692901611 disk.dm-1.out‚ 10.6640625! disk.dm-0.out‚1458.6806640625 disk.rbd1.out‚1.390625& disk.nvme0c0n1.out‚1458.6806640625 disk.sda1.in‚0( proc.memory.availableMB‚ 495.91796875 disk.nvme0n1.in‚ 560.734375$ disk.nvme0n1.out‚1458.6806640625 disk.dm-0.in‚ 560.734375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 560.734375 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 10.6640625* proc.memory.percent‚16.263341903686523£Ù”ˆ‚(:‰ ßñìÒØÉ“í gpu.0.gpu‚91.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚232.459' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.4918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§9©a¸‰‚(:­ ßñìÒìô²ï disk.dm-1.out‚ 11.00390625 disk.dm-0.in‚ 560.734375 network.recv‚181003 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda1.out‚0 cpu‚25.846981439649497 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.390625 disk.sda.out‚ 11.00390625 disk.sda1.in‚0" proc.memory.rssMB‚ 2664.5859375" disk.dm-0.out‚1462.94873046875* disk./.usagePercent‚39.556651886771434% disk.nvme0n1.out‚1462.94873046875 disk.sda3.in‚0* proc.memory.percent‚16.263341903686523 network.sent‚782344 disk.sda2.out‚0' disk.nvme0c0n1.out‚1462.94873046875 disk.dm-1.in‚0$ disk./.usageGB‚589.4226989746094 disk.nvme0n1.in‚ 560.734375 disk.nvme0c0n1.in‚ 560.734375$ memory_percent‚96.97329998016357( proc.memory.availableMB‚ 495.89453125 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 11.00390625 proc.cpu.threads‚17é5ðÙ0Š+ _wandbruntime‚518  _runtime‚518õó’=Ä‹¾ & lr-AdamW/pg1‚0.0005818746863056092 & lr-AdamW/pg2‚0.0005818746863056092 trainer/global_step‚599 % _timestamp‚1.7843632398407993e+09 _runtime‚ 518.346742042 _step‚45–8èȌ trainer/global_step‚599 % _timestamp‚1.7843632398407993e+09 _runtime‚ 518.346742042 _step‚45 & lr-AdamW/pg1‚0.0005818746863056092 & lr-AdamW/pg2‚0.0005818746863056092-“î0+ _wandbruntime‚518  _runtime‚518Ηµº»'ŽµO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626670205440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20831616496640 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90476041323093 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 627011543552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703726 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5520533 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90310025412608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 % _timestamp‚1.7843632398418653e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90310008373248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 trainer/global_step‚599 _step‚46 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92772352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4703726 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111384576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89850350153728 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90937036956160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 _runtime‚ 518.348014173 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90936678578688 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90937036956160 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89850333114368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 627011543552 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5530508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1286541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20197127481344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90310025412608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1055401 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1055398 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90310008373248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634489015296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728P¡Çqý!¹O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚90310008373248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 634489015296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20831727881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4703728 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚90936678578688 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 626670205440 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚20831616496640 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90476041323093 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 625708208725 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚533 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90476398912335 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚10224259 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 627011543552 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4703726 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 626048758607 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5520533 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚90310025412608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 % _timestamp‚1.7843632398418653e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚90310008373248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 trainer/global_step‚599 _step‚46 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 634581787648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92772352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4703726 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111384576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89850350153728 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚90937036956160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 _runtime‚ 518.348014173 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚90936678578688 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2341677 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚90937036956160 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚89850333114368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4703728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 627011543552 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5530508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 A7DeviceStatsMonitor.on_train_batch_starûÿN¯Ât/active.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5530508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1286541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20197127481344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚90310025412608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1055401 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1055398 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1.œÝ ×0+ _wandbruntime‚518  _runtime‚5184¨×1Å&‘¿M E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4711358 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10250934 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚599 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 _runtime‚ 518.975368259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91085290619904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1057143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20230556239872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89996843786240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843632404690955e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 627682832384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5529585 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90623920638041 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 635508988928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 _step‚47 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0jv«6É&’ÃM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89996843786240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843632404690955e+09 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 627682832384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5529585 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚90623920638041 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚20866172411904 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 635508988928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2345473 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107183104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 _step‚47 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4711358 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4711358 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10250934 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5529585 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚599 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 628024177664 _runtime‚ 518.975368259 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91085290619904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1057143 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20230556239872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0/3À†l0“+ _wandbruntime‚518  _runtime‚5184]_õü”ö 0loss/FBt1/assignment_loss‚0.1168405358187512 0loss/FBt2/assignment_loss‚0.4760332929803745 .loss/FRt2/detection_loss‚1.487438132882902 1loss/SRqqt2/detection_loss‚0.6512644573553241 %loss/total_loss‚18.61208053824278 /loss/FRt1/detection_loss‚0.6539724114835587 -loss/FBt2/detection_loss‚2.25537730474025 _runtime‚ 518.976298304 _step‚48 2loss/SRqqt1/assignment_loss‚1.5123760079344113 1loss/SRqqt1/detection_loss‚2.4228893643865983 trainer/global_step‚599 % _timestamp‚1.7843632404703863e+09 /loss/FRt1/assignment_loss‚3.565720799634623 /loss/FBt1/detection_loss‚1.5797622622525587 epoch‚1 /loss/FRt2/assignment_loss‚2.564908004091383 1loss/SRqqt2/assignment_loss‚1.325497964682045 ê*«•ú epoch‚1 /loss/FRt2/assignment_loss‚2.564908004091383 1loss/SRqqt2/assignment_loss‚1.325497964682045 0loss/FBt1/assignment_loss‚0.1168405358187512 0loss/FBt2/assignment_loss‚0.4760332929803745 .loss/FRt2/detection_loss‚1.487438132882902 1loss/SRqqt2/detection_loss‚0.6512644573553241 %loss/total_loss‚18.61208053824278 /loss/FRt1/detection_loss‚0.6539724114835587 -loss/FBt2/detection_loss‚2.25537730474025 _runtime‚ 518.976298304 _step‚48 2loss/SRqqt1/assignment_loss‚1.5123760079344113 1loss/SRqqt1/detection_loss‚2.4228893643865983 trainer/global_step‚599 % _timestamp‚1.7843632404703863e+09 /loss/FRt1/assignment_loss‚3.565720799634623 /loss/FBt1/detection_loss‚1.57976226225255870ïVz”–‚(:‰ îñìҺȬï gpu.0.gpu‚57.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/KÔ´¸—‚(:­ îñìÒ¥ïÑð" disk.dm-0.out‚1469.24365234375 disk.rbd1.in‚0 disk.rbd1.out‚1.453125 disk.sda.out‚ 11.41015625* proc.memory.percent‚16.263341903686523 disk.dm-0.in‚ 560.734375 disk.sda3.out‚ 11.41015625 disk.dm-1.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 11.41015625* disk./.usagePercent‚39.556647534643346 disk.sda2.out‚0' disk.nvme0c0n1.out‚1469.24365234375 disk.sda.in‚0 cpu‚25.843304417658953 network.recv‚184632$ disk./.usageGB‚589.4226341247559 disk.nvme0c0n1.in‚ 560.734375 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2664.5859375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 560.734375 network.sent‚800685 disk.rbd0.out‚0( proc.memory.availableMB‚ 495.37890625 proc.cpu.threads‚17 disk.sda3.in‚0% disk.nvme0n1.out‚1469.24365234375$ memory_percent‚96.97644710540771q±­˜‚(:’ ýñìÒ•ñþî gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X=/Ó±™‚(:¦ ýñìÒ¡ÃÃñ) disk./.usagePercent‚39.55660759746801& disk.nvme0c0n1.out‚1472.3759765625 disk.sda2.out‚0" proc.memory.rssMB‚ 2664.5859375 proc.cpu.threads‚17 network.recv‚189483 disk.dm-1.in‚0 disk.dm-1.out‚ 11.83203125 disk.drbd2165.out‚0$ memory_percent‚96.97465896606445& proc.memory.availableMB‚ 495.671875 disk.sda1.out‚0 disk.sda3.out‚ 11.83203125 disk.rbd0.out‚0 disk.dm-0.in‚ 560.734375 disk.rbd1.out‚1.453125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.263341903686523 disk.sda3.in‚0 disk.nvme0n1.in‚ 560.734375$ disk./.usageGB‚589.4220390319824 disk.sda.out‚ 11.83203125$ disk.nvme0n1.out‚1472.3759765625 disk.sda2.in‚0 cpu‚25.83941421713121 disk.nvme0c0n1.in‚ 560.734375 disk.sda.in‚0! disk.dm-0.out‚1472.3759765625 network.sent‚852764YŽ}È”š‚(:‰ ŒòìÒâÐïî gpu.0.gpu‚87.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚40.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)<—è±›‚(:¦ ŒòìÒõ”Šñ disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 616.39453125 disk.sda1.in‚0 disk.sda3.out‚ 12.22265625 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 616.39453125 network.recv‚192966$ disk./.usageGB‚589.4218673706055 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.53125 disk.drbd2165.out‚0 disk.dm-0.in‚ 616.39453125 disk.dm-0.out‚ 1479.53515625* proc.memory.percent‚16.263341903686523 disk.sda.in‚0 network.sent‚858011 disk.rbd0.in‚0 cpu‚25.8354176197786 disk.sda.out‚ 12.22265625" disk.nvme0n1.out‚ 1479.53515625 proc.cpu.threads‚17( proc.memory.availableMB‚ 495.91015625 disk.sda2.in‚0$ memory_percent‚96.97320461273193 disk.sda1.out‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 2664.5859375$ disk.nvme0c0n1.out‚ 1479.53515625) disk./.usagePercent‚39.55659607712897 disk.dm-1.out‚ 12.22265625p?ÖÞ0œ+ _wandbruntime‚559  _runtime‚559VyDBľ % _timestamp‚1.7843632808903341e+09 _runtime‚ 559.396323388 _step‚49 & lr-AdamW/pg1‚0.0005817909678731406 & lr-AdamW/pg2‚0.0005817909678731406 trainer/global_step‚649IÃ`Ȟ & lr-AdamW/pg1‚0.0005817909678731406 & lr-AdamW/pg2‚0.0005817909678731406 trainer/global_step‚649 % _timestamp‚1.7843632808903341e+09 _runtime‚ 559.396323388 _step‚491ß"]e0Ÿ+ _wandbruntime‚559  _runtime‚559¡  µO _step‚50 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98349783939584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22554317548032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1142648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97852191403601 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98349425562112 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 685647047168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5984960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843632808914435e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97672059854848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 677707045376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1389953 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1142651 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2532338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97175866707968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97672059854848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5984960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 UGDeviceStatsMonitor.on_train_batch_\›Ùs°start/active_bytes.small_pool.current‚ 341338112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98349783939584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97672076894208 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 trainer/global_step‚649 _runtime‚ 559.397735895 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 676324695633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2532604 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 677365707264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120(™|ä¿'¡¹O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 676324695633 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 677365707264 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚22554212454912 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685733528064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5085328 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2532604 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 677365707264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 _step‚50 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98349783939584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22554317548032 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1142648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97852191403601 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98349425562112 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 685647047168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5085326 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5984960 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1389690 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105093120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5085328 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843632808914435e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98349425562112 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97672059854848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 677707045376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1389953 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1142651 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2532338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚97175866707968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97672059854848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5984960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98349783939584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97672076894208 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚97852548992843 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 trainer/global_step‚649 _runtime‚ 559.3977358952Š•îÏ0¢+ _wandbruntime‚560  _runtime‚560š[11Ä&£¾M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1391722 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 686674672640 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2536368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5994026 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092960 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 677676299349 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97819300884480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚649 $ _timestamp‚1.784363281520158e+09 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98498037603328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21902012778496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97322394419200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 678719679488 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 678719679488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 678378334208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5984037 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1144396 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1144396 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102988800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2536368 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5092960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5994026 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2536115 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5092960 [JDeviceStatsMon w/ÈN itor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 677676299349 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97819300884480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚649 $ _timestamp‚1.784363281520158e+09 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 686759049216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98498037603328 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚97819317923840 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21902012778496 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97322394419200 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚11693ð£ð 0¥+ _wandbruntime‚560  _runtime‚560Gé¦ú 1loss/SRqqt1/assignment_loss‚1.618130414688494 trainer/global_step‚649 % _timestamp‚1.7843632815220337e+09 _step‚52 0loss/FRt1/assignment_loss‚3.4714341177259174 .loss/FRt2/detection_loss‚1.510079298945687 1loss/SRqqt1/detection_loss‚2.1572216199710965 /loss/FBt1/detection_loss‚1.0455044211922497 %loss/total_loss‚17.06441477216807 /loss/FRt2/assignment_loss‚2.574682897130648 2loss/SRqqt2/assignment_loss‚1.3782397173431422 1loss/FBt1/assignment_loss‚0.10120297163574925 /loss/FRt1/detection_loss‚0.6380410057524485 _runtime‚ 560.027743423 0loss/FBt2/assignment_loss‚0.3375971242113055 1loss/SRqqt2/detection_loss‚0.7469090959421432 /loss/FBt2/detection_loss‚1.4853720876291907 epoch‚1ÊmA§þ trainer/global_step‚649 % _timestamp‚1.7843632815220337e+09 _step‚52 0loss/FRt1/assignment_loss‚3.4714341177259174 .loss/FRt2/detection_loss‚1.510079298945687 1loss/SRqqt1/detection_loss‚2.1572216199710965 /loss/FBt1/detection_loss‚1.0455044211922497 %loss/total_loss‚17.06441477216807 /loss/FRt2/assignment_loss‚2.574682897130648 2loss/SRqqt2/assignment_loss‚1.3782397173431422 1loss/FBt1/assignment_loss‚0.10120297163574925 /loss/FRt1/detection_loss‚0.6380410057524485 _runtime‚ 560.027743423 0loss/FBt2/assignment_loss‚0.3375971242113055 1loss/SRqqt2/detection_loss‚0.7469090959421432 /loss/FBt2/detection_loss‚1.4853720876291907 epoch‚1 1loss/SRqqt1/assignment_loss‚1.6181304146884944o’¨‚(:‡ ›òìÒ—ŸÍï gpu.0.gpu‚95.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓÑÃ*º©‚(:¯ ›òìÒñôñ disk.rbd1.out‚1.53125% disk.nvme0n1.out‚1483.65966796875" memory_percent‚96.974778175354 disk.rbd1.in‚0 network.recv‚196808" disk.nvme0c0n1.in‚ 616.39453125 disk.sda.in‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2664.5859375 cpu‚25.831086415153653) disk./.usagePercent‚39.55662500598034 disk.dm-0.in‚ 616.39453125 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 12.51171875 network.sent‚874416" disk.dm-0.out‚1483.65966796875 disk.sda1.out‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 616.39453125* proc.memory.percent‚16.263341903686523 disk.sda2.out‚0 disk.sda.out‚ 12.51171875 disk.drbd2165.out‚0( proc.memory.availableMB‚ 495.65234375$ disk./.usageGB‚589.4222984313965' disk.nvme0c0n1.out‚1483.65966796875 disk.sda3.out‚ 12.51171875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17M–·tŸª‚(:” ªòìÒŠý¹í gpu.0.gpu‚87.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚218.193' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.638600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÄTa·«‚(:¬ ªòìÒñÞ‰ï cpu‚25.828112179222256 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 616.39453125 disk.rbd1.in‚0# disk./.usageGB‚589.422420501709 disk.dm-1.out‚ 12.7734375 disk.sda2.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚1489.4482421875 disk.sda2.out‚0* proc.memory.percent‚16.263365745544434 network.sent‚925283 network.recv‚202526 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚1489.4482421875 disk.sda.in‚0$ memory_percent‚96.97554111480713# proc.memory.rssMB‚ 2664.58984375 disk.nvme0n1.in‚ 616.39453125$ disk.nvme0n1.out‚1489.4482421875 disk.sda.out‚ 12.7734375 proc.cpu.threads‚17( proc.memory.availableMB‚ 495.52734375* disk./.usagePercent‚39.556633198221434 disk.drbd2165.out‚0 disk.rbd1.out‚1.65625" disk.nvme0c0n1.in‚ 616.39453125 disk.sda3.out‚ 12.7734375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0—oXžž¬‚(:“ ¹òìÒŠÁñì gpu.0.gpu‚88.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚39.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æJG\¼­‚(:± ¹òìÒëŽâï network.sent‚932603# disk./.usageGB‚589.422290802002 disk.sda3.out‚ 13.11328125 disk.dm-1.in‚0 disk.dm-1.out‚ 13.11328125 disk.rbd0.out‚0% disk.nvme0n1.out‚1493.10595703125 disk.dm-0.in‚ 616.39453125 disk.sda1.out‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 616.39453125$ memory_percent‚96.97558879852295 cpu‚25.824976549339983 disk.nvme0n1.in‚ 616.39453125 network.recv‚206724 disk.sda.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚1493.10595703125) disk./.usagePercent‚39.55662449396527 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.65625* proc.memory.percent‚16.263365745544434 disk.rbd0.in‚0' disk.nvme0c0n1.out‚1493.10595703125 proc.cpu.threads‚17 disk.sda.out‚ 13.11328125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 495.51953125# proc.memory.rssMB‚ 2664.58984375ïpç0®+ _wandbruntime‚600  _runtime‚600é¼cŸÄ¯¾ _step‚53 & lr-AdamW/pg1‚0.0005816885732095646 & lr-AdamW/pg2‚0.0005816885732095646 trainer/global_step‚699 % _timestamp‚1.7843633219175315e+09 _runtime‚ 600.423705987L›~È°Â & lr-AdamW/pg1‚0.0005816885732095646 & lr-AdamW/pg2‚0.0005816885732095646 trainer/global_step‚699 % _timestamp‚1.7843633219175315e+09 _runtime‚ 600.423705987 _step‚535Ù'aþ0±+ _wandbruntime‚600  _runtime‚600ö²ðíÇ'²ÁO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105762172545536 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6439412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1493176 _runtime‚ 600.424972208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104501400301568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1229898 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6429437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105762530923008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728402547200 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23540003334144 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727281732423 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚699 % _timestamp‚1.7843633219187903e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚54 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105034128375808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2723077 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104501417340928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99063808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1229901 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11896363 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105034111336448 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728402547200 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23540021946368 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2722813 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882QÊÀßË'³ÅO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5466926 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117676032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24276918988288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚54 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105034128375808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2723077 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104501417340928 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99063808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1229901 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11896363 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105034111336448 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 728402547200 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23540021946368 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105762172545536 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2722813 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 736897041920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105762172545536 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6439412 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 728061209088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1493176 _runtime‚ 600.424972208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104501400301568 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1229898 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6429437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105762530923008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 728402547200 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5466926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23540003334144 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 727281732423 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚699 % _timestamp‚1.7843633219187903e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9975 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6438489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105910426202112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24311252244480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 729073836032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@¾Å]xDeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 trainer/global_step‚699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23573432092672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728292786257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚55 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1494941 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1231646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚ 601.0569429 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6438489 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263?“ •Ò&¶ÌM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5474558 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5474560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6448478 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23573432092672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 728292786257 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105181369405440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚55 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 729073836032 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1494941 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2726584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1231646 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚ 601.0569429 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9989 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6438489 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 727952236263 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6438489 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105910426202112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24311252244480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 729073836032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 737910819840 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105910784586752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 trainer/global_step‚699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 trainer/global_step‚749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25999451414528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6893864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777557669449 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25211459872768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113175277906432 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1317151 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778756710912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112396179857408 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113175277906432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112396162818048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 _step‚58 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78092288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96704512 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 788088246272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779098049024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2913220 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0YÒÀßÓÃÃO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5848528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1596072 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 128324096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112604491564617 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778756710912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚112396179857408 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778756710912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113175277906432 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112396162818048 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 _step‚58 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78092288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96704512 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 788088246272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589242 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 779098049024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2913220 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25211441260544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112604849153859 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111826933895168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6893864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112396179857408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377472 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5848526 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779098049024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚ 641.222057187 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113174919528960 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 trainer/global_step‚749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚25999451414528 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6893864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 777557669449 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25211459872768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12099 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113175277906432 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6883889 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1317151 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360:‘Æ„0Ä+ _wandbruntime‚641  _runtime‚641úHXžÑ&ÅËM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1598110 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5856160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112543420887040 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112752370879565 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 778909273165 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚749 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112752013290211 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779769337856 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6892941 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 789116360192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113323531570176 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5856158 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112543420887040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12759090 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26034004991488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112543403847680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚ 641.848740273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2917274 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1318896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1598378 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚26033897808384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113323531570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843633633425295e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111973461606400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5856160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5856160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 778909273165 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5856158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 778568723171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚749 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚112752013290211 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 779769337856 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6902930 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6892941 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 789116360192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113323531570176 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5856158 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1318893 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112543420887040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6892941 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12759090 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779769337856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2917003 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚26034004991488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚112543403847680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358384640 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 _runtime‚ 641.848740273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088;>Ûš?0Ç+ _wandbruntime‚641  _runtime‚641‘âþøÈý /loss/FRt1/assignment_loss‚3.246269948664896 1loss/FBt2/assignment_loss‚0.13201330417756585 1loss/SRqqt1/detection_loss‚1.6871163254476966 1loss/SRqqt2/detection_loss‚0.7647663941573095 &loss/total_loss‚15.849242835105853 epoch‚1 % _timestamp‚1.7843633633441935e+09 1loss/SRqqt1/assignment_loss‚1.549714367154069 trainer/global_step‚749 2loss/SRqqt2/assignment_loss‚1.2313591934168462 /loss/FRt1/detection_loss‚0.6767951883725407 /loss/FRt2/detection_loss‚1.5258130568811288 _runtime‚ 641.849870267 0loss/FRt2/assignment_loss‚2.3161860704421997 1loss/FBt1/assignment_loss‚0.10222649001573143 /loss/FBt1/detection_loss‚0.7997636085021976 /loss/FBt2/detection_loss‚1.8172188878736713 _step‚60\—T'É 0loss/FRt2/assignment_loss‚2.3161860704421997 1loss/FBt1/assignment_loss‚0.10222649001573143 /loss/FBt1/detection_loss‚0.7997636085021976 /loss/FBt2/detection_loss‚1.8172188878736713 _step‚60 /loss/FRt1/assignment_loss‚3.246269948664896 1loss/FBt2/assignment_loss‚0.13201330417756585 1loss/SRqqt1/detection_loss‚1.6871163254476966 1loss/SRqqt2/detection_loss‚0.7647663941573095 &loss/total_loss‚15.849242835105853 epoch‚1 % _timestamp‚1.7843633633441935e+09 1loss/SRqqt1/assignment_loss‚1.549714367154069 trainer/global_step‚749 2loss/SRqqt2/assignment_loss‚1.2313591934168462 /loss/FRt1/detection_loss‚0.6767951883725407 /loss/FRt2/detection_loss‚1.5258130568811288 _runtime‚ 641.849870267ÍÍ‚(:¦ õòìÒ—ªï disk.rbd0.out‚0$ disk./.usageGB‚589.4256744384766 disk.sda1.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚709.25& proc.memory.availableMB‚ 11.1484375* proc.memory.percent‚16.263365745544434 disk.sda2.in‚0 disk.dm-1.out‚ 14.12890625 disk.dm-0.in‚709.25 network.recv‚223295 disk.sda2.out‚0 disk.sda.out‚ 14.12890625" disk.dm-0.out‚1512.17236328125* disk./.usagePercent‚39.556851572648135 disk.rbd1.out‚1.796875 disk.sda.in‚0 disk.sda3.out‚ 14.12890625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚1512.17236328125 disk.nvme0c0n1.in‚709.25 disk.dm-1.in‚0 cpu‚25.813®nÈ6ä330806700574 proc.cpu.threads‚17# proc.memory.rssMB‚ 2664.58984375$ memory_percent‚99.93195533752441 network.sent‚ 1.066868e+06 disk.rbd1.in‚0' disk.nvme0c0n1.out‚1512.17236328125 disk.drbd2165.in‚0øl+äŸÎ‚(:” „óìҳ݀ð gpu.0.gpu‚94.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚133.889' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.777800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B©N®Ï‚(:£ „óìÒÔªò network.recv‚226916% disk.nvme0n1.out‚1517.00341796875 disk.nvme0c0n1.in‚709.25 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.875 disk.sda.out‚ 14.46484375$ memory_percent‚99.93212223052979 network.sent‚ 1.074082e+06 disk.rbd0.out‚0# proc.memory.rssMB‚ 2664.58984375* proc.memory.percent‚16.263365745544434 cpu‚25.811027333023926 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 11.12109375 disk.sda3.in‚0 disk.sda3.out‚ 14.46484375 disk.rbd0.in‚0 disk.dm-0.in‚709.25* disk./.usagePercent‚39.556698736150196 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚1517.00341796875 disk.dm-1.in‚0 disk.nvme0n1.in‚709.25" disk.dm-0.out‚1517.00341796875 disk.sda2.out‚0 proc.cpu.threads‚17# disk./.usageGB‚589.423397064209 disk.dm-1.out‚ 14.46484375¥á}ž0Ð+ _wandbruntime‚682  _runtime‚682öPÄѾ % _timestamp‚1.7843634038626022e+09 _runtime‚ 682.368711537 _step‚61 & lr-AdamW/pg1‚0.0005814277826946801 & lr-AdamW/pg2‚0.0005814277826946801 trainer/global_step‚799VæÍ0ÈÒ & lr-AdamW/pg1‚0.0005814277826946801 & lr-AdamW/pg2‚0.0005814277826946801 trainer/global_step‚799 % _timestamp‚1.7843634038626022e+09 _runtime‚ 682.368711537 _step‚61= üö%0Ó+ _wandbruntime‚682  _runtime‚682›€&-Ç'ÔÁO ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120588024889856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3103509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1404401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 trainer/global_step‚799 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3103234 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119758214299648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26882879186944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6230126 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 682.369949254 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119152467488768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107190272 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 828174156357 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6230126 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6230128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27722236320256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119758231339008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7348316 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119758214299648É`dË'ÕÅO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119758231339008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 828514706239 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12099 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1404401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 trainer/global_step‚799 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3103234 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120587666512384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119758214299648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26882879186944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549882 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6230126 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88578048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚ 682.369949254 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚119152467488768 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26882897799168 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1404398 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107190272 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 828174156357 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056583 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338112 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6230126 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757120 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6230128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27722236320256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119980641645125 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119758231339008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7338341 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948352 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7348316 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119758214299648 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 829793550848 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120588024889856 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3103509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948352 IºÜ20Ö+ _wandbruntime‚682  _runtime‚682ðu£íÏ&×ÉM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120735920168960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120128520960073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 829525760073 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120736278553600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1406146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3107277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119298995200000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830464839680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119905472368640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚63 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 682.99934792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119905455329280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 830806184960 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7347393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6237760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6237758 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1701131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 840359241728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7357382 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27756685799424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840274865152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 830806184960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7357382 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 $ _timestamp‚1.784363404493206e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26916326557696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119298978160640ê´9ŒþØÍM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589354 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 829525760073 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚120®¼©ôÕ%736278553600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12099 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1406146 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549994 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3107277 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119298995200000 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84376576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 830464839680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚119905472368640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9989 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚63 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚ 682.99934792 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119905455329280 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 830806184960 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358384640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967975 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7347393 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 128324096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119905472368640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚119905455329280 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6237760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6237758 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1701131 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 840359241728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7357382 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345280 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26916307945472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27756685799424 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 840274865152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6237758 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 830806184960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7357382 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056583 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 $ _timestamp‚1.784363404493206e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26916326557696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚119298978160640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120735920168960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚120128520960073 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3107007 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169?¤ ˪0Ù+ _wandbruntime‚683  _runtime‚683û=ÑKÚú 2loss/SRqqt1/assignment_loss‚1.1954755678164715 /loss/FBt1/detection_loss‚0.9534061515213362 .loss/FBt2/detection_loss‚2.060989930982823 % _timestamp‚1.7843634044948297e+09 /loss/FRt1/assignment_loss‚3.206055077317105 /loss/FRt2/assignment_loss‚2.324934288833271 1loss/FBt2/assignment_loss‚0.06317101266573079 epoch‚1 _step‚64 1loss/SRqqt2/assignment_loss‚1.215361071699702 _runtime‚ 683.000515534 1loss/FBt1/assignment_loss‚0.11528344690644342 /loss/FRt1/detection_loss‚0.6918166513903465 .loss/FRt2/detection_loss‚1.436112466604235 1loss/SRqqt1/detection_loss‚1.5483220965050313 1loss/SRqqt2/detection_loss‚0.7491220514889926 &loss/total_loss‚15.560049813731489 trainer/global_step‚799ùÃ"Ûþ 1loss/FBt2/assignment_loss‚0.06317101266573079 epoch‚1 _step‚64 1loss/SRqqt2/assignment_loss‚1.215361071699702 _runtime‚ 683.000515534 1loss/FBt1/assignment_loss‚0.11528344690644342 /loss/FRt1/detection_loss‚0.6918166513903465 .loss/FRt2/detection_loss‚1.436112466604235 1loss/SRqqt1/detection_loss‚1.5483220965050313 1loss/SRqqt2/detection_loss‚0.7491220514889926 &loss/total_loss‚15.560049813731489 trainer/global_step‚799 2loss/SRqqt1/assignment_loss‚1.1954755678164715 /loss/FBt1/detection_loss‚0.9534061515213362 .loss/FBt2/detection_loss‚2.060989930982823 % _timestamp‚1.7843634044948297e+09 /loss/FRt1/assignment_loss‚3.206055077317105 /loss/FRt2/assignment_loss‚2.324934288833271@´ƒž8’Ü‚(:‡ “óìÒÏ»çï gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,5³Ý‚(:¨ “óìÒ³¯óñ disk.dm-0.in‚ 776.5546875$ memory_percent‚99.96035099029541# proc.memory.rssMB‚ 2717.58203125 network.sent‚ 1.077004e+06$ disk./.usageGB‚589.4234161376953! disk.nvme0c0n1.in‚ 776.5546875 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 776.5546875# disk.nvme0n1.out‚1519.748046875 disk.rbd1.out‚1.875 disk.sda1.out‚0& proc.memory.availableMB‚ 6.49609375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.out‚ 14.7109375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚0* proc.memory.percent‚16.586804389953613 cpu‚25.845104649244227 network.recv‚230556% disk.nvme0c0n1.out‚1519.748046875 disk.sda.out‚ 14.7109375) disk./.usagePercent‚39.55670001618787 disk.sda1.in‚0 disk.sda3.out‚ 14.7109375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚1519.748046875 disk.sda2.out‚0 disk.rbd1.in‚0ÞYž0Þ+ _wandbruntime‚700  _runtime‚700½°wåØ=ßÒ{ & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4588119387626648 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45678675174713135 /particle/accuracy_2_of_2‚0.9998962879180908 particle/accuracy_2_of_4‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4645833373069763 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0069974311627447605 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5002086162567139 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +jet/accuracy_2_of_2‚0.09726011008024216 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4342556595802307 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4066310524940491 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666728258133 +jet/accuracy_2_of_4‚0.46423155069351196 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5338541865348816 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0019101311918348074 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4588119387626648 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.03014797531068325 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.481008380651474 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8496546745300293 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.832945704460144 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4580061733722687 +jet/accuracy_3_of_4‚0.07019110023975372 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13059747219085693 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42376503348350525 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_3‚0.32418984174728394 particle/accuracy_2_of_3‚1 /particle/accuracy_3_of_3‚0.9990025758743286 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.42461514472961426 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4865844249725342 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4632776379585266 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5073863863945007 +particle/sensitivity‚0.5629563927650452 *loss/val_total_loss‚14.811808617838405 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928713440895081 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10232003778219223 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4444444477558136 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4484192430973053 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7021920084953308 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48454052209854126 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909065961837769 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2578125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0869511067867279 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 &particle/f_score‚0.572712242603302 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9943965673446655 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5630348324775696 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 /particle/accuracy_4_of_4‚0.9061880707740784 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005670979619026184 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2079160064458847 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.46034759283065796 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6803099513053894 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7110921740531921 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.45712411403656006 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 epoch‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49855858087539673 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41972193121910095 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5840054750442505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.024015437811613083 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/specificity‚0.8898041248321533 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887730479240417 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6606764793395996 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.11401668190956116 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09390036016702652 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09874628484249115 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.05956965312361717 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971235394477844 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46166056394577026 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48250943422317505 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26291951537132263 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43183550238609314 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883680939674377 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05325939506292343 jet/accuracy_4_of_5‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6120936870574951 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0034938014578074217 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09020515531301498 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.04526643082499504 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963691234588623 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4721325933933258 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.5733246803283691 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.030425528064370155 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47529610991477966 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44958123564720154 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05808132141828537 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4513092041015625 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.031359560787677765 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09120529890060425 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5300846099853516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6808506846427917 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07369954884052277 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.983015775680542 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_1_of_1‚0.19896109402179718 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09099972248077393 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5027849078178406 *jet/accuracy_1_of_4‚0.7220582365989685 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5332449674606323 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.44597747921943665 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.04445565491914749 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.031359560787677765 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.051788851618766785 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.04318879544734955 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.04462440684437752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07499619573354721 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.013467980548739433 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.19439104199409485 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.06684718281030655 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.031258925795555115 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4648648798465729 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4484192430973053 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.07181054353713989 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09919828176498413 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852653741836548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7validation_average_jet_accuracy‚0.23910625278949738 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.045346930623054504 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744776487350464 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2001221477985382 *jet/accuracy_1_of_3‚0.7093461751937866 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.25925925374031067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09390036016702652 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08861693739891052 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996309757232666 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7"Purity/*FRt*SRqqt*FBt/event_purity‚0.17424276471138 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950128197669983 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41056686639785767 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 jet/accuracy_5_of_5‚0 _step‚65 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941112995147705 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3329295217990875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.422626256942749 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03453482314944267 'particle/accuracy‚0.819611132144928 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.26637884974479675 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872387647628784 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 trainer/global_step‚799 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43204325437545776 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8251244425773621 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920659065246582 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877501726150513 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6873960494995117 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8823810815811157 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_3‚0.05310776084661484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4657992720603943 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34302979707717896 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.993103563785553 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42957377433776855 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4789915084838867 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6073384284973145 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9170374870300293 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4855799376964569 /particle/accuracy_3_of_4‚0.9974937438964844 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 _runtime‚ 700.323631953 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915801286697388 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014275765046477318 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10332866758108139 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996309757232666 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 % _timestamp‚1.7843634218169498e+09¹V#bkàÖ{ =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.45712411403656006 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 epoch‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49855858087539673 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.41972193121910095 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5840054750442505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.024015437811613083 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +particle/specificity‚0.8898041248321533 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9887730479240417 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6606764793395996 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_1_of_1‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.11401668190956116 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1388888955116272 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09390036016702652 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09874628484249115 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.05956965312361717 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971235394477844 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46166056394577026 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48250943422317505 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26291951537132263 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.43183550238609314 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9883680939674377 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.05325939506292343 jet/accuracy_4_of_5‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6120936870574951 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0034938014578074217 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09020515531301498 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.04526643082499504 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963691234588623 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4721325933933258 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_2‚0.5733246803283691 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.030425528064370155 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.47529610991477966 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.44958123564720154 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.05808132141828537 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4513092041015625 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.031359560787677765 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09120529890060425 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5300846099853516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6808506846427917 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07369954884052277 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0¨‡7€q-.983015775680542 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_1_of_1‚0.19896109402179718 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1388888955116272 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09099972248077393 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5027849078178406 *jet/accuracy_1_of_4‚0.7220582365989685 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5332449674606323 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.44597747921943665 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.04445565491914749 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.031359560787677765 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.051788851618766785 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.04318879544734955 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.04462440684437752 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07499619573354721 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.013467980548739433 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.19439104199409485 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.06684718281030655 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.031258925795555115 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4648648798465729 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4484192430973053 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.07181054353713989 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09919828176498413 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9852653741836548 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7validation_average_jet_accuracy‚0.23910625278949738 9 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.045346930623054504 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744776487350464 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2001221477985382 *jet/accuracy_1_of_3‚0.7093461751937866 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.25925925374031067 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09390036016702652 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08861693739891052 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996309757232666 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7"Purity/*FRt*SRqqt*FBt/event_purity‚0.17424276471138 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950128197669983 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41056686639785767 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 jet/accuracy_5_of_5‚0 _step‚65 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941112995147705 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3329295217990875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.422626256942749 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03453482314944267 'particle/accuracy‚0.819611132144928 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.26637884974479675 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872387647628784 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 trainer/global_step‚799 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43204325437545776 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8251244425773621 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920659065246582 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_2‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877501726150513 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6873960494995117 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8823810815811157 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_3‚0.05310776084661484 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4657992720603943 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34302979707717896 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.993103563785553 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42957377433776855 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4789915084838867 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6073384284973145 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9170374870300293 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4855799376964569 /particle/accuracy_3_of_4‚0.9974937438964844 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 _runtime‚ 700.323631953 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9915801286697388 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.014275765046477318 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10332866758108139 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996309757232666 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 % _timestamp‚1.7843634218169498e+09 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4588119387626648 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45678675174713135 /particle/accuracy_2_of_2‚0.9998962879180908 particle/accuracy_2_of_4‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4645833373069763 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0069974311627447605 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5002086162567139 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +jet/accuracy_2_of_2‚0.09726011008024216 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4342556595802307 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4066310524940491 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0716666728258133 +jet/accuracy_2_of_4‚0.46423155069351196 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5338541865348816 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0019101311918348074 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4588119387626648 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.03014797531068325 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.481008380651474 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8496546745300293 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.832945704460144 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4580061733722687 +jet/accuracy_3_of_4‚0.07019110023975372 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.13059747219085693 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42376503348350525 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_2_of_3‚0.32418984174728394 particle/accuracy_2_of_3‚1 /particle/accuracy_3_of_3‚0.9990025758743286 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.42461514472961426 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4865844249725342 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4632776379585266 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5073863863945007 +particle/sensitivity‚0.5629563927650452 *loss/val_total_loss‚14.811808617838405 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9928713440895081 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10232003778219223 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4444444477558136 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4484192430973053 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7021920084953308 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48454052209854126 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909065961837769 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2578125 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0869511067867279 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 &particle/f_score‚0.572712242603302 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9943965673446655 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5630348324775696 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 /particle/accuracy_4_of_4‚0.9061880707740784 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005670979619026184 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2079160064458847 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.46034759283065796 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6803099513053894 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7110921740531921 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_5‚1A¦¡­’á‚(:‡ ¢óìÒ¨¼ºð gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚256.11' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c’Gµâ‚(:ª ¢óìÒ…¡õò disk.sda1.in‚0) disk./.usagePercent‚39.57284692338747 disk.sda.in‚0$ disk./.usageGB‚589.6640167236328 disk.dm-1.in‚0 disk.dm-0.in‚ 776.5546875 disk.sda.out‚ 15.0078125) proc.memory.percent‚16.64876937866211 disk.sda3.in‚0 disk.nvme0n1.in‚ 776.5546875! disk.nvme0c0n1.in‚ 776.5546875# memory_percent‚96.9567060470581! proc.memory.rssMB‚ 2727.734375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 15.0078125$ disk.nvme0n1.out‚1768.9716796875& disk.nvme0c0n1.out‚1768.9716796875( proc.memory.availableMB‚ 498.61328125 disk.rbd0.in‚0! disk.dm-0.out‚1768.9716796875 cpu‚25.892403838058453 network.sent‚ 1.152732e+06 network.recv‚234911 disk.sda2.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 15.0078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚1.9375´±Ù”ã‚(:‰ ±óìÒѸ­ï gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚248.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.6088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i:a͹ä‚(:® ±óìÒå’åñ disk.sda.out‚ 15.42578125 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 15.42578125! disk.dm-0.out‚1804.2236328125) disk./.usagePercent‚39.57277421724775$ disk./.usageGB‚589.6629333496094 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 776.5546875# proc.memory.rssMB‚ 2728.20703125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 776.5546875 cpu‚25.88894685728961 network.sent‚ 1.19919e+06 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚1.9375 disk.rbd0.in‚0( proc.memory.availableMB‚ 498.33984375) proc.memory.percent‚16.65165424346924 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 15.42578125 disk.drbd2165.out‚0 disk.dm-0.in‚ 776.5546875$ disk.nvme0n1.out‚1804.2236328125$ memory_percent‚96.95837497711182 network.recv‚239521& disk.nvme0c0n1.out‚1804.2236328125úû1å‚(:’ ÀóìÒõŒ®í gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚214.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚42.932199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t¾Àæ‚(:µ ÀóìÒõðÒï" disk.dm-0.out‚2023.52783203125 disk.rbd1.out‚2.015625 disk.sda3.out‚ 15.77734375) proc.memory.percent‚16.65165424346924$ memory_percent‚96.95839881896973 network.recv‚242862 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 776.5546875 disk.sda2.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 498.3359375* disk./.usagePercent‚39.572786249601855$ disk./.usageGB‚589.6631126403809 disk.sda.out‚ 15.77734375 disk.dm-1.in‚0 disk.dm-0.in‚ 776.5546875! disk.nvme0c0n1.in‚ 776.5546875' disk.nvme0c0n1.out‚2023.52783203125 network.sent‚ 1.202154e+06 disk.sda3.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 15.77734375 disk.sda2.in‚0# proc.memory.rssMB‚ 2728.20703125 cpu‚25.885039922740194 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚0% disk.nvme0n1.out‚2023.52783203125®·–Ù0ç+ _wandbruntime‚742  _runtime‚742è•cÄè¾ & lr-AdamW/pg1‚0.0005812694035934589 & lr-AdamW/pg2‚0.0005812694035934589 trainer/global_step‚849 % _timestamp‚1.7843634642960587e+09 _runtime‚ 742.802036752 _step‚66Ñ¿ýÈé & lr-AdamW/pg2‚0.0005812694035934589 trainer/global_step‚849 % _timestamp‚1.7843634642960587e+09 _runtime‚ 742.802036752 _step‚66 & lr-AdamW/pg1‚0.0005812694035934589B´ÀP30ê+ _wandbruntime‚742  _runtime‚7426‹Æ'ëÀO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1494171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129049832431616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3329418 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1835250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28570229213184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129283174664939 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128386664296448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129948071729664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897880920064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129049832431616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129947713351680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129049849470976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7873091 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29478818041344 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 908588828160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 896850918377 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 _runtime‚ 742.803399666 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6732392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1835518 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28570247825408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128386681335808 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1494168 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6732394 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 908666919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 % _timestamp‚1.7843634642972324e+09 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3329689 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6732394 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7863115 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6732394 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 908666919936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 % _timestamp‚1.7843634642972324e+09 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3329689 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6732394 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7863115 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129049832431616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3329418 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1835250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28570229213184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129283174664939 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128386664296448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129948071729664 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 897880920064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129049832431616 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129947713351680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 C:DeviceStatsMonitor.on_train_batch_starýjÕY9t/active.all.current‚9978 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129049849470976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7873091 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29478818041344 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 908588828160 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚29478914745344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 896850918377 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 898222258688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 _runtime‚ 742.803399666 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6732392 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129947713351680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1835518 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7863115 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28570247825408 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128386681335808CF«V0í+ _wandbruntime‚743  _runtime‚743î8|ÁÑ&îËM WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129431053979891 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚68 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29513369557504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6740026 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1495913 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7882157 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899234892800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28603657971712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129197090500608 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898893547008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 909692973568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129197073461248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6740026 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28603676583936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128533192007680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130095967008256 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6740024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1837584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29513262374912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100ûßšÕ&ïÏM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129197073461248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6740026 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28603676583936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚128533192007680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚130095967008256 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6740024 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1837584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚29513262374912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129431053979891 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚68 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129197090500608 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29513369557504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6740026 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1495913 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7882157 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3333228 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 899234892800 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 909604403200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28603657971712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129197090500608 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 898893547008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 909692973568 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 899234892800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0D˜H0ð+ _wandbruntime‚743  _runtime‚743[Œrñü 0loss/FRt2/assignment_loss‚2.2745390354300574 2loss/SRqqt1/assignment_loss‚1.2882621754609531 $ _timestamp‚1.784363464929591e+09 1loss/SRqqt2/detection_loss‚0.8095655123722384 /loss/FBt2/detection_loss‚1.3656171393891177 &loss/total_loss‚14.307789455424604 _runtime‚ 743.435269162 0loss/FRt1/assignment_loss‚3.1518012759662715 2loss/SRqqt2/assignment_loss‚1.1187695260207677 1loss/FBt1/assignment_loss‚0.13754282708469234 1loss/FBt2/assignment_loss‚0.05847716964679724 /loss/FRt1/detection_loss‚0.6306857001408787 trainer/global_step‚849 _step‚69 /loss/FRt2/detection_loss‚1.4343418705841633 /loss/SRqqt1/detection_loss‚1.39416024478069 /loss/FBt1/detection_loss‚0.6440269785479784 epoch‚2˜·¶Ðò€ $ _timestamp‚1.784363464929591e+09 1loss/SRqqt2/detection_loss‚0.8095655123722384 /loss/FBt2/detection_loss‚1.3656171393891177 &loss/total_loss‚14.307789455424604 _runtime‚ 743.435269162 0loss/FRt1/assignment_loss‚3.1518012759662715 2loss/SRqqt2/assignment_loss‚1.1187695260207677 1loss/FBt1/assignment_loss‚0.13754282708469234 1loss/FBt2/assignment_loss‚0.05847716964679724 /loss/FRt1/detection_loss‚0.6306857001408787 trainer/global_step‚849 _step‚69 /loss/FRt2/detection_loss‚1.4343418705841633 /loss/SRqqt1/detection_loss‚1.39416024478069 /loss/FBt1/detection_loss‚0.6440269785479784 epoch‚2 0loss/FRt2/assignment_loss‚2.2745390354300574 2loss/SRqqt1/assignment_loss‚1.2882621754609531Ež‚|žó‚(:“ ÏóìÒè¹Ëí gpu.0.gpu‚86.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.905' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.98099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^ê@€¾ô‚(:³ ÏóìÒÑѲï disk.sda2.out‚0" disk.nvme0c0n1.in‚ 842.94140625 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 1.207072e+06) disk./.usagePercent‚39.57281953058131 disk.sda.in‚0 disk.sda3.out‚ 15.96484375 disk.dm-1.out‚ 15.96484375 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚2026.77294921875% proc.memory.availableMB‚ 497.53125) proc.memory.percent‚16.65167808532715 disk.rbd1.in‚0 disk.rbd1.out‚2.015625 disk.sda1.out‚0 disk.nvme0n1.in‚ 842.94140625 disk.dm-0.in‚ 842.94140625 cpu‚25.88115199658613$ memory_percent‚96.96331024169922 network.recv‚246326$ disk./.usageGB‚589.6636085510254 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚2026.77294921875 disk.sda.out‚ 15.96484375' disk.nvme0c0n1.out‚2026.77294921875 disk.drbd2165.in‚0 disk.sda3.in‚0" proc.memory.rssMB‚ 2728.2109375vâ6É“õ‚(:ˆ ÞóìÒÏÎòî gpu.0.gpu‚13.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÙÇcÀö‚(:µ ÞóìÒ‰†„ñ disk.sda3.out‚ 16.32421875 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 842.94140625' disk.nvme0c0n1.out‚2031.65966796875" proc.memory.rssMB‚ 2728.2109375 disk.sda3.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚2031.65966796875" disk.dm-0.out‚2031.65966796875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 16.32421875 disk.nvme0n1.in‚ 842.94140625 disk.rbd1.out‚2.078125 disk.sda1.in‚0 network.sent‚ 1.271715e+06( proc.memory.availableMB‚ 497.88671875# disk./.usageGB‚589.663631439209 disk.drbd2165.in‚0 disk.sda.out‚ 16.32421875# memory_percent‚96.9611406326294) proc.memory.percent‚16.65167808532715 network.recv‚249977) disk./.usagePercent‚39.57282106662651 disk.dm-0.in‚ 842.94140625 disk.sda.in‚0 cpu‚25.878232519326676 disk.sda1.out‚0 proc.cpu.threads‚17G«¾»÷‚(:’ íóìÒËÏ‹ï gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘…ãÀø‚(:µ íóìÒñ‘Âñ$ disk./.usageGB‚589.6634788513184 disk.dm-1.in‚0 disk.dm-1.out‚ 16.5546875) proc.memory.percent‚16.65167808532715% disk.nvme0n1.out‚2037.41845703125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 16.5546875 disk.sda3.in‚0 disk.sda2.in‚0 network.sent‚ 1.275892e+06 disk.sda3.out‚ 16.5546875 disk.rbd0.in‚0 disk.rbd1.out‚2.078125( proc.memory.availableMB‚ 497.99609375 disk.drbd2165.out‚0" disk.dm-0.out‚2037.41845703125" disk.nvme0c0n1.in‚ 842.94140625 disk.sda1.out‚0" proc.memory.rssMB‚ 2728.2109375 disk.nvme0n1.in‚ 842.94140625 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚96.96047306060791 proc.cpu.threads‚17 disk.sda2.out‚0' disk.nvme0c0n1.out‚2037.41845703125 network.recv‚253605* disk./.usagePercent‚39.572810826325146 disk.dm-0.in‚ 842.94140625 disk.rbd1.in‚0 cpu‚25.874567938573648'¶˜v0ù+ _wandbruntime‚783  _runtime‚7833î”Äú¾ & lr-AdamW/pg1‚0.0005810923817602297 & lr-AdamW/pg2‚0.0005810923817602297 trainer/global_step‚899 % _timestamp‚1.7843635053538828e+09 _runtime‚ 783.860893805 _step‚70QÈû _step‚70 & lr-AdamW/pg1‚0.0005810923817602297 & lr-AdamW/pg2‚0.0005810923817602297 trainer/global_step‚899 % _timestamp‚1.7843635053538828e+09 _runtime‚ 783.860893805F—–ta0ü+ _wandbruntime‚783  _runtime‚783Ÿ‰T C ý¿O HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚136659324745447 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8317567 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚137360460335104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1939563 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚899 % _timestamp‚1.7843635053546827e+09 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31201613707264 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚136659682334693 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135712197890048 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1581421 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚136411900952576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 _step‚71 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136411883913216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948576421888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8317567 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31201712508416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3520726 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7113994 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136411900952576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137360818713088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7113994 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚136411900952576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137360460335104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 _step‚71 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚136411883913216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 948576421888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30241685751808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8317567 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31201712508416 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3520726 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7113994 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8327543 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚136411900952576 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7113992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚137360818713088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7113994 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136411883913216 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7113992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 948917760512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚899 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3524760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121624 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135858725601280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 $ _timestamp‚1.784363505982402e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1941335 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961056742400 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7121626 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚136559141982208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚72 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137509072376832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136559141982208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136559124942848 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136807204060399 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8326619 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15448243 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1941594 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31236171252736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 949589048832 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8336609 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278912 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587Ñ@G’ÎM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7121624 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135858725601280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 $ _timestamp‚1.784363505982402e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7121624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1941335 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1437 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 961056742400 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7121626 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚136559141982208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _step‚72 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1583166 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚137509072376832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚136559141982208 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 948137909121 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136559124942848 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3524498 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚136807204060399 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8326619 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15448243 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1941594 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚30275114510336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31236171252736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 949589048832 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8336609 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 949930394624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278912 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚136806846471041 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚899 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3524760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0H2&R|0‚+ _wandbruntime‚784  _runtime‚784˜úƒý /loss/FRt1/detection_loss‚0.6143052438232182 /loss/FRt2/detection_loss‚1.4319640891380498 $ _timestamp‚1.784363505983986e+09 _step‚73 0loss/FRt2/assignment_loss‚2.2218653571627396 2loss/SRqqt2/assignment_loss‚1.1296106746192767 1loss/FBt2/assignment_loss‚0.13428139681801504 /loss/FBt1/detection_loss‚0.8881555921899821 &loss/total_loss‚14.937418192187012 2loss/SRqqt1/assignment_loss‚1.1420409714325248 0loss/FBt1/assignment_loss‚0.1323530605704445 /loss/FBt2/detection_loss‚1.6754788697830267 epoch‚2 1loss/SRqqt1/detection_loss‚1.5559713931763983 1loss/SRqqt2/detection_loss‚0.8188313685272258 trainer/global_step‚899 _runtime‚ 784.490000869 0loss/FRt1/assignment_loss‚3.1925601749461094¥ß}„ 0loss/FRt1/assignment_loss‚3.1925601749461094 /loss/FRt1/detection_loss‚0.6143052438232182 /loss/FRt2/detection_loss‚1.4319640891380498 $ _timestamp‚1.784363505983986e+09 _step‚73 0loss/FRt2/assignment_loss‚2.2218653571627396 2loss/SRqqt2/assignment_loss‚1.1296106746192767 1loss/FBt2/assignment_loss‚0.13428139681801504 /loss/FBt1/detection_loss‚0.8881555921899821 &loss/total_loss‚14.937418192187012 2loss/SRqqt1/assignment_loss‚1.1420409714325248 0loss/FBt1/assignment_loss‚0.1323530605704445 /loss/FBt2/detection_loss‚1.6754788697830267 epoch‚2 1loss/SRqqt1/detection_loss‚1.5559713931763983 1loss/SRqqt2/detection_loss‚0.8188313685272258 trainer/global_step‚899 _runtime‚ 784.490000869IDꤔ…‚(:‰ üóìÒö”ùí gpu.0.gpu‚86.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Ñ™EÁ†‚(:¶ üóìÒÑÌÿï disk.sda1.out‚0$ memory_percent‚96.95885181427002( proc.memory.availableMB‚ 498.26171875 cpu‚25.87116882480942) disk./.usagePercent‚39.57282081061898 disk.nvme0n1.in‚ 842.94140625 disk.dm-0.in‚ 842.94140625 disk.sda3.in‚0 disk.dm-1.out‚ 17.07421875 disk.drbd2165.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚589.6636276245117 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 network.recv‚256899% disk.nvme0n1.out‚2040.55419921875 disk.rbd1.out‚2.15625* proc.memory.percent‚16.649770736694336 disk.rbd0.in‚0 disk.sda.out‚ 17.07421875" proc.memory.rssMB‚ 2727.8984375 disk.sda3.out‚ 17.07421875 disk.drbd2165.out‚0" disk.dm-0.out‚2040.55419921875" disk.nvme0c0n1.in‚ 842.94140625 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.278856e+06 disk.dm-1.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚2040.55419921875]A4Í”‡‚(:‰ ‹ôìÒöÌ´î gpu.0.gpu‚95.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úòì ²ˆ‚(:§ ‹ôìÒÝÑ—ð disk.dm-0.in‚ 899.59375 disk.sda1.in‚0) proc.memory.percent‚16.65167808532715 proc.cpu.threads‚17$ disk./.usageGB‚589.6636390686035 disk.sda.in‚0 disk.sda.out‚ 17.390625 network.sent‚ 1.343818e+06 disk.nvme0n1.in‚ 899.59375 disk.nvme0c0n1.in‚ 899.59375" proc.memory.rssMB‚ 2728.2109375 cpu‚25.867781019285044 disk.dm-1.out‚ 17.390625 disk.rbd0.in‚0" disk.dm-0.out‚2045.74951171875 disk.rbd1.in‚0 disk.rbd1.out‚2.15625 disk.sda3.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚2045.74951171875' disk.nvme0c0n1.out‚2045.74951171875 disk.sda1.out‚0 disk.sda3.out‚ 17.390625 disk.sda2.out‚0$ memory_percent‚96.96123600006104) disk./.usagePercent‚39.57282157864158 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 497.87109375 network.recv‚261726 disk.sda2.in‚0 disk.dm-1.in‚0ì@Ð0‰+ _wandbruntime‚824  _runtime‚824ú¥9ÓŠ¼ % lr-AdamW/pg2‚0.000580896728564774 trainer/global_step‚949 % _timestamp‚1.7843635461580849e+09 _runtime‚ 824.664201627 _step‚74 % lr-AdamW/pg1‚0.000580896728564774Qep­Æ‹À % _timestamp‚1.7843635461580849e+09 _runtime‚ 824.664201627 _step‚74 % lr-AdamW/pg1‚0.000580896728564774 % lr-AdamW/pg2‚0.000580896728564774 trainer/global_step‚949J ̬Ç0Œ+ _wandbruntime‚824  _runtime‚824¸ü©øÈ'ÂO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1668671 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1668668 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚32924541836800 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843635461593437e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2043414 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3711810 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚75 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144773207318528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8772019 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7495592 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 999613262336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32924632249344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31913123678208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143037748523008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143773952434176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71800320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997743342307 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1011508571136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 999271923712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998083892193 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8781995 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 trainer/global_step‚949 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1011436770816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495594 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8772019 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143773935394816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143773935394816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2043414 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3711810 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2043142 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚90412544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚75 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 999271923712 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7495592 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144773207318528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8772019 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7495592 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 999613262336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚32924632249344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31913123678208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚143037748523008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144035474825955 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144773565696512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚143773952434176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚71800320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997743342307 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1011508571136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 999271923712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144773207318528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998083892193 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8781995 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 trainer/global_step‚949 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1011436770816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7495594 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8772019 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143773935394816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143773935394816 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3712085 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 998754396029 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚949 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8791061 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1000284550656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2045437 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8791061 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144921460975104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32959095808512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144182996551549 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1670416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144183354140907 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999094946027 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1000625896448 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8781071 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7503226 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143921193463808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31946533824512 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7503226 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143921176424448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1000625896448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1012456898560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 998754396029 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚949 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8791061 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1000284550656 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2045437 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1670413 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8791061 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144921460975104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1000284550656 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3715593 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚32959095808512 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144182996551549 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1670416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144183354140907 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999094946027 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1000625896448 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8781071 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7503226 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143921193463808 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚32958990723072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31946533824512LQ‚F»0”+ _wandbruntime‚825  _runtime‚825ö¸ßX•ý &loss/total_loss‚14.141115224031825 % _timestamp‚1.7843635467919896e+09 _step‚77 1loss/FBt1/assignment_loss‚0.10284461483033261 1loss/SRqqt2/detection_loss‚0.8202635510061339 /loss/FRt1/assignment_loss‚3.038667739729194 0loss/SRqqt1/detection_loss‚1.510276206156468 epoch‚2 _runtime‚ 825.297685237 /loss/FBt2/detection_loss‚1.3231033622287214 2loss/SRqqt1/assignment_loss‚1.3353437229512848 1loss/SRqqt2/assignment_loss‚1.003463635327924 1loss/FBt2/assignment_loss‚0.32834292747584204 /loss/FRt1/detection_loss‚0.6039203886875534 /loss/FRt2/detection_loss‚1.4713921706176645 0loss/FBt1/detection_loss‚0.47155584682290586 trainer/global_step‚949 0loss/FRt2/assignment_loss‚2.1319410581977998° ü– _step‚77 1loss/FBt1/assignment_loss‚0.10284461483033261 1loss/SRqqt2/detection_loss‚0.8202635510061339 /loss/FRt1/assignment_loss‚3.038667739729194 0loss/SRqqt1/detection_loss‚1.510276206156468 epoch‚2 _runtime‚ 825.297685237 /loss/FBt2/detection_loss‚1.3231033622287214 2loss/SRqqt1/assignment_loss‚1.3353437229512848 1loss/SRqqt2/assignment_loss‚1.003463635327924 1loss/FBt2/assignment_loss‚0.32834292747584204 /loss/FRt1/detection_loss‚0.6039203886875534 /loss/FRt2/detection_loss‚1.4713921706176645 0loss/FBt1/detection_loss‚0.47155584682290586 trainer/global_step‚949 0loss/FRt2/assignment_loss‚2.1319410581977998 &loss/total_loss‚14.141115224031825 % _timestamp‚1.7843635467919896e+09Mðäbª’—‚(:‡ ©ôìÒè¯ùî gpu.0.gpu‚86.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¸cU²˜‚(:§ ©ôìÒôëßï disk.sda1.in‚0 disk.sda3.out‚ 18.04296875 disk.rbd0.out‚0 disk.dm-0.in‚ 899.59375$ memory_percent‚96.95754051208496) proc.memory.percent‚16.65170192718506 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 cpu‚25.861988233755294 disk.drbd2165.in‚0 disk.sda2.in‚0 network.sent‚ 1.398946e+06 disk.dm-1.out‚ 18.04296875# disk.nvme0n1.out‚2054.158203125 disk.dm-0.out‚2054.158203125 disk.rbd1.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 498.4765625# proc.memory.rssMB‚ 2728.21484375 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 899.59375 disk.sda.out‚ 18.04296875 network.recv‚269536) disk./.usagePercent‚39.57281671449843 disk.sda3.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚589.6635665893555 disk.nvme0n1.in‚ 899.59375 disk.rbd1.out‚2.21875% disk.nvme0c0n1.out‚2054.158203125ÓžŸ™‚(:” ¸ôìÒîø¢ï gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R^üªš‚(:Ÿ ¸ôìÒIJÖñ* disk./.usagePercent‚39.572810570317614$ disk./.usageGB‚589.6634750366211 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-0.out‚ 2057.1328125$ memory_percent‚96.95754051208496' proc.memory.availableMB‚ 498.4765625 proc.cpu.threads‚17 network.recv‚273433 disk.sda1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.out‚ 2057.1328125) proc.memory.percent‚16.65170192718506 disk.dm-1.in‚0 disk.dm-1.out‚ 18.6484375 disk.nvme0c0n1.in‚ 899.59375 disk.sda.out‚ 18.6484375# proc.memory.rssMB‚ 2728.21484375 cpu‚25.85996078669562 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 899.59375 disk.sda2.out‚0 disk.sda3.out‚ 18.6484375 disk.rbd0.in‚0! disk.nvme0n1.out‚ 2057.1328125 network.sent‚ 1.451192e+06 disk.drbd2165.out‚0 disk.dm-0.in‚ 899.59375 disk.rbd1.in‚0 disk.rbd1.out‚2.296875ÜÍó©0›+ _wandbruntime‚865  _runtime‚865Ÿ#·ÊÄœ¾ & lr-AdamW/pg2‚0.0005806824565735307 trainer/global_step‚999 % _timestamp‚1.7843635871728504e+09 _runtime‚ 865.679834717 _step‚78 & lr-AdamW/pg1‚0.0005806824565735307íÎëÈ _step‚78 & lr-AdamW/pg1‚0.0005806824565735307 & lr-AdamW/pg2‚0.0005806824565735307 trainer/global_step‚999 % _timestamp‚1.7843635871728504e+09 _runtime‚ 865.679834717Nlí¾0ž+ _wandbruntime‚865  _runtime‚865n¡=¡#ŸÈO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9226471 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98801152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚34647500849152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 _step‚79 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9226471 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151135986876416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1062939244544 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1755918 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚150363265077248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34647402048000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2147259 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1062859055616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151136003915776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2147539 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1048700379101 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9236447 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚152186312679936 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7877192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3903460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚152185954301952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7877194 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151411624906463 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9236447 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151135986876416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 % _timestamp‚1.7843635871735945e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1050308764160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33584542992384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1048359829215 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1050308764160 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7877192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1049967425536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1049967425536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150363282116608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 trainer/global_step‚999 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33584561604608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17103663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_star6“-t/reserved_bytes.large_pool.freed‚0 _runtime‚ 865.679980802 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚151411624906463 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9236447 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151135986876416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1755921 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 % _timestamp‚1.7843635871735945e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1050308764160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚33584542992384 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1048359829215 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1050308764160 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7877192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1049967425536 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1049967425536 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚150363282116608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 trainer/global_step‚999 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7877194 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33584561604608 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17103663 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚ 865.679980802 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3903177 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152185954301952 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚34647402048000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151136003915776 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2147259 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1062859055616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151136003915776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360O ß0¡+ _wandbruntime‚866  _runtime‚866[TzsØ&¢ÒM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150509775749120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33617990363136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33617971750912 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 866.310999296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151283227906048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3907254 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1757663 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1063968337408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151559504221415 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1049711432935 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151283227906048 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1051321398272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚999 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152334566343680 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9245513 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 % _timestamp‚1.7843635878048346e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9245513 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34681958700544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34681859906560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063888155648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9235523 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1757666 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7884824 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1051321398272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152334207958528 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚999 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152334566343680 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9245513 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 % _timestamp‚1.7843635878048346e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9245513 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚34681958700544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚34681859906560 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1063888155648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9235523 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1757666 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7884824 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚150509775749120 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33617990363136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7884824 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33617971750912 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 866.310999296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151283227906048 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3907254 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1757663 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1063968337408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚151559504221415 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1049711432935 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3907001 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0P €öÈ0¤+ _wandbruntime‚866  _runtime‚866N<“~ÿ¥ù $ _timestamp‚1.784363587806596e+09 _runtime‚ 866.312285865 0loss/FBt1/assignment_loss‚0.1124856526705065 1loss/SRqqt2/detection_loss‚0.7778496767251404 .loss/FBt1/detection_loss‚0.624423469209128 /loss/FBt2/detection_loss‚1.3587702992372215 2loss/SRqqt1/assignment_loss‚1.3308625446984896 2loss/SRqqt2/assignment_loss‚1.0188990382826173 0loss/SRqqt1/detection_loss‚1.538772820782017 trainer/global_step‚999 _step‚81 /loss/FRt1/assignment_loss‚3.025901261943838 1loss/FBt2/assignment_loss‚0.11509652315561336 epoch‚2 /loss/FRt2/assignment_loss‚2.094833220171135 /loss/FRt1/detection_loss‚0.6451568818103228 /loss/FRt2/detection_loss‚1.5371142402191058 &loss/total_loss‚14.180165628905137ö=‚/¶¦ý 0loss/SRqqt1/detection_loss‚1.538772820782017 trainer/global_step‚999 _step‚81 /loss/FRt1/assignment_loss‚3.025901261943838 1loss/FBt2/assignment_loss‚0.11509652315561336 epoch‚2 /loss/FRt2/assignment_loss‚2.094833220171135 /loss/FRt1/detection_loss‚0.6451568818103228 /loss/FRt2/detection_loss‚1.5371142402191058 &loss/total_loss‚14.180165628905137 $ _timestamp‚1.784363587806596e+09 _runtime‚ 866.312285865 0loss/FBt1/assignment_loss‚0.1124856526705065 1loss/SRqqt2/detection_loss‚0.7778496767251404 .loss/FBt1/detection_loss‚0.624423469209128 /loss/FBt2/detection_loss‚1.3587702992372215 2loss/SRqqt1/assignment_loss•H XM‚1.3308625446984896 2loss/SRqqt2/assignment_loss‚1.0188990382826173Q}a”§‚(:‰ ÇôìÒ”âãî gpu.0.gpu‚46.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚36.0 gpu.0.powerWatts‚277.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ­ ¯¨‚(:¤ ÇôìÒÞéÂï disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 2089.42578125# proc.memory.rssMB‚ 2728.21484375) disk./.usagePercent‚39.62454226876908 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 977.78125 cpu‚25.85684969650344 disk.sda.out‚ 18.87890625 disk.sda3.out‚ 18.87890625 proc.cpu.threads‚17 network.sent‚ 1.456142e+06 disk.sda2.in‚0 disk.rbd1.out‚2.296875 disk.dm-1.in‚0 disk.rbd0.out‚0# memory_percent‚96.9580888748169( proc.memory.availableMB‚ 498.38671875$ disk./.usageGB‚590.4343147277832 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 977.78125 disk.dm-1.out‚ 18.87890625 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 977.78125 disk.dm-0.out‚ 2089.42578125 network.recv‚277077" disk.nvme0n1.out‚ 2089.42578125 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.percent‚16.65170192718506 disk.rbd0.in‚0† ¤Ã”©‚(:‰ ÖôìÒ븄ï gpu.0.gpu‚89.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚256.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.2336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#)›¸ª‚(:­ ÖôìÒ¥³¤ï network.recv‚281232) disk./.usagePercent‚39.62273331953214 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 977.78125' disk.nvme0c0n1.out‚2092.31494140625 disk.rbd1.out‚2.359375 disk.sda1.out‚0& proc.memory.availableMB‚ 498.359375 proc.cpu.threads‚17 disk.rbd0.out‚0$ disk./.usageGB‚590.4073600769043 disk.sda3.out‚ 19.19140625% disk.nvme0n1.out‚2092.31494140625 disk.dm-0.in‚ 977.78125 disk.sda.out‚ 19.19140625 cpu‚25.854003235844644 disk.drbd2165.out‚0$ memory_percent‚96.95825576782227 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 network.sent‚ 1.518949e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚ 977.78125" disk.dm-0.out‚2092.31494140625 disk.sda1.in‚0# proc.memory.rssMB‚ 2728.21484375) proc.memory.percent‚16.65170192718506 disk.dm-1.out‚ 19.19140625 disk.rbd0.in‚0_X”«‚(:‰ åôìÒÛš¥ï gpu.0.gpu‚94.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚38.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àk¢'µ¬‚(:ª åôìÒÖó×ñ disk.rbd1.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚2885.0166015625 disk.sda2.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚589.6634750366211 network.sent‚ 1.523857e+06( proc.memory.availableMB‚ 498.36328125 disk.sda1.in‚0 disk.dm-1.out‚ 19.4453125 disk.nvme0n1.in‚ 977.78125 disk.sda2.in‚0$ memory_percent‚96.95823192596436 cpu‚25.851094727766323 network.recv‚284829# proc.memory.rssMB‚ 2728.21484375& disk.nvme0c0n1.out‚2885.0166015625 disk.sda.in‚0 disk.sda.out‚ 19.4453125 disk.drbd2165.out‚0 disk.dm-0.in‚ 977.78125 disk.rbd1.out‚2.359375 disk.sda1.out‚0 disk.sda3.out‚ 19.4453125 disk.drbd2165.in‚0 disk.rbd0.in‚0) proc.memory.percent‚16.65170192718506* disk./.usagePercent‚39.572810570317614 disk.nvme0c0n1.in‚ 977.78125 disk.sda3.in‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚2885.0166015625Ä0­+ _wandbruntime‚906  _runtime‚906½b 6Å®¿ & lr-AdamW/pg1‚0.0005804495795487897 & lr-AdamW/pg2‚0.0005804495795487897 trainer/global_step‚1049 % _timestamp‚1.7843636282591639e+09 _runtime‚ 906.765118708 _step‚82¡BËnɯà _runtime‚ 906.765118708 _step‚82 & lr-AdamW/pg1‚0.0005804495795487897 & lr-AdamW/pg2‚0.0005804495795487897 trainer/global_step‚1049 % _timestamp‚1.7843636282591639e+09RÑ'Û0°+ _wandbruntime‚906  _runtime‚906ÏÅTÐ'±ÊO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9680923 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 906.766363417 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8258794 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35255999531008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1100662927360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35255980918784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158788132576217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159599059663360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843636282602286e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36370350418432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158498055397376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159598701285376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101004265984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1843171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157688815710208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1843168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4094227 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159599059663360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2251327 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1098976316123 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158498055397376 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9680923 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157688798670848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17939715 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258792 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4094498 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚83 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8258792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1114350887424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 trainer/global_step‚1049VrÒ0Ô'²ÎO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1843171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157688815710208 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159598701285376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚268 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9690899 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1843168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102995456 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1101004265984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4094227 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159599059663360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2251059 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1114266504192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2251327 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1098976316123 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158498055397376 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9690899 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9680923 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157688798670848 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17939715 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8258792 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4094498 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚83 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158498038358016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8258792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1114350887424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 trainer/global_step‚1049 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9680923 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1100662927360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 906.766363417 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158498038358016 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8258794 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35255999531008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1100662927360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚35255980918784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158788132576217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159599059663360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7843636282602286e+09 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36370350418432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚158787774986971 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚158498055397376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159598701285376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1101004265984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0S{|IÀ0³+ _wandbruntime‚907  _runtime‚907%ÄÓ°´ÓM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2253143 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158935296712565 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17966391 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35289409677312 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 907.397660203 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158645296427008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157835326382080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 trainer/global_step‚1049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9689975 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4098336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9699965 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8266424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1844913 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8266424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159746954941952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1844916 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36404811935744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1115305561600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843636288914661e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056 bODeviceStatsMonitor.on_train_batG'77) ch_end/inactive_split_bytes.large_pool.allocated‚35289428289536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158645296427008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1102016900096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1101675554304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159747313327104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9689975 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158645279387648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100327919843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990ü|DÝ&µ×M N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36404811935744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1115305561600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843636288914661e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4098056 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35289428289536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158645296427008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1102016900096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 562036736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1101675554304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159747313327104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9689975 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158645279387648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1100327919843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 562036736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159747313327104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2253143 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚158935296712565 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17966391 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1101675554304 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696832 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8266426 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35289409677312 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚ 907.397660203 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚158645296427008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157835326382080 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚157835309342720 trainer/global_step‚1049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9689975 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4098336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚280 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36404715238912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9699965 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8266424 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1115383646208 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1437 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1844913 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8266424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159746954941952 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1844916 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0TË\¥×0¶+ _wandbruntime‚907  _runtime‚907!±Ñ©ÿ·ù 2loss/SRqqt2/assignment_loss‚1.0199417889028914 1loss/FBt2/assignment_loss‚0.21699544450327973 /loss/FBt1/detection_loss‚0.5676394104858419 _runtime‚ 907.398819572 /loss/FRt2/assignment_loss‚2.046115697104976 /loss/FRt2/detection_loss‚1.5348903797979836 1loss/SRqqt1/detection_loss‚1.3447742342877274 /loss/FBt2/detection_loss‚2.3818095311936407 $loss/total_loss‚14.9796924028452 .loss/FRt1/assignment_loss‚3.07538172406085 1loss/FBt1/assignment_loss‚0.17945660702162963 /loss/FRt1/detection_loss‚0.6413477275871666 0loss/SRqqt2/detection_loss‚0.817014981486034 epoch‚2 % _timestamp‚1.7843636288931432e+09 1loss/SRqqt1/assignment_loss‚1.154324876413179 trainer/global_step‚1049 _step‚85Ò88{¸ý _runtime‚ 907.398819572 /loss/FRt2/assignment_loss‚2.046115697104976 /loss/FRt2/detection_loss‚1.5348903797979836 1loss/SRqqt1/detection_loss‚1.3447742342877274 /loss/FBt2/detection_loss‚2.3818095311936407 $loss/total_loss‚14.9796924028452 .loss/FRt1/assignment_loss‚3.07538172406085 1loss/FBt1/assignment_loss‚0.17945660702162963 /loss/FRt1/detection_loss‚0.6413477275871666 0loss/SRqqt2/detection_loss‚0.817014981486034 epoch‚2 % _timestamp‚1.7843636288931432e+09 1loss/SRqqt1/assignment_loss‚1.154324876413179 trainer/global_step‚1049 _step‚85 2loss/SRqqt2/assignment_loss‚1.0199417889028914 1loss/FBt2/assignment_loss‚0.21699544450327973 /loss/FBt1/detection_loss‚0.5676394104858419Uò.ç㔹‚(:‰ ôôìҴݸï gpu.0.gpu‚45.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ǘ˜©º‚(:ž ôôìÒ”¿òñ) disk./.usagePercent‚39.57101646951766 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚2.5# proc.memory.rssMB‚ 2728.21484375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.65170192718506 disk.nvme0n1.in‚ 977.78125$ disk./.usageGB‚589.6367416381836 disk.dm-1.in‚0$ memory_percent‚96.96018695831299 disk.dm-1.out‚19.78125! disk.dm-0.out‚2888.5693359375 disk.sda.in‚0 disk.sda.out‚19.78125 cpu‚25.849203360594483 network.recv‚288180 disk.rbd0.out‚0( proc.memory.availableMB‚ 498.04296875 network.sent‚ 1.528817e+06 disk.sda3.out‚19.78125 disk.dm-0.in‚ 977.78125 disk.sda2.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚2888.5693359375 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 977.78125& disk.nvme0c0n1.out‚2888.5693359375 •´ž»‚(:“ ƒõìÒÈâ©í gpu.0.gpu‚35.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.876' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.975199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÚI´¼‚(:© ƒõìÒ©¥¯ï disk.dm-1.out‚ 20.1953125 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2728.21484375 disk.drbd2165.out‚0" disk.dm-0.out‚2895.24462890625 disk.sda1.in‚0' proc.memory.availableMB‚ 498.0078125% disk.nvme0n1.out‚2895.24462890625 disk.rbd1.out‚2.5 cpu‚25.84647793795132 disk.sda3.out‚ 20.1953125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 1027.84375) proc.memory.percent‚16.65170192718506* disk./.usagePercent‚39.570991124771766 disk.dm-1.in‚0$ memory_percent‚96.96040153503418 network.recv‚292752$ disk./.usageGB‚589.6363639831543 disk.nvme0n1.in‚ 1027.84375 disk.sda.out‚ 20.1953125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1027.84375' disk.nvme0c0n1.out‚2895.24462890625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.in‚0 network.sent‚ 1.591731e+06 disk.sda3.in‚0WØЖ”½‚(:‰ ’õìÒÔ÷êì gpu.0.gpu‚87.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÆ﻾‚(:° ’õìÒä«Ýð disk.nvme0n1.in‚ 1027.84375% disk.nvme0n1.out‚2898.84326171875* disk./.usagePercent‚39.570990100741625 disk.dm-1.in‚0 disk.dm-0.in‚ 1027.84375" disk.dm-0.out‚2898.84326171875 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚ 20.41015625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 1027.84375' disk.nvme0c0n1.out‚2898.84326171875$ memory_percent‚96.96154594421387' proc.memory.availableMB‚ 497.8203125 disk.sda3.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2728.21484375 network.sent‚ 1.598644e+06 network.recv‚295969 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚2.5625$ disk./.usageGB‚589.6363487243652 disk.sda3.out‚ 20.41015625 disk.dm-1.out‚ 20.41015625 disk.drbd2165.out‚0 disk.sda1.in‚0) proc.memory.percent‚16.65170192718506 cpu‚25.844137292117974 disk.sda.in‚0Y.wi0¿+ _wandbruntime‚947  _runtime‚947{L„óÅÀ¿ & lr-AdamW/pg1‚0.0005801981124478072 & lr-AdamW/pg2‚0.0005801981124478072 trainer/global_step‚1099 % _timestamp‚1.7843636691860735e+09 _runtime‚ 947.692003205 _step‚86@CÒÉÁà & lr-AdamW/pg1‚0.0005801981124478072 & lr-AdamW/pg2‚0.0005801981124478072 trainer/global_step‚1099 % _timestamp‚1.7843636691860735e+09 _runtime‚ 947.692003205 _step‚86V(Id0Â+ _wandbruntime‚947  _runtime‚947u|Ô'ÃÎO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38093078987264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7843636691871116e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4286246 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38093194565632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1149933352917 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1151358429184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166164282656725 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1151699767808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1930421 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1930418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10135375 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10145351 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 _runtime‚ 947.696804587 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167011448268800 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36927418845184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165014332264448 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640394 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640394 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115578368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167011806646784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚87 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2355825 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1165660142080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚1099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165860106878976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166163925067479 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165860089839616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2355541 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1149592803031 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165860089839616 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165860106878976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224¥‘I®±ÄÒO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚38093194565632 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1149933352917 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24404557824 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1437 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1151358429184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166164282656725 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚268 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554757632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1151699767808 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1437 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1930421 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1930418 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24404557824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 562036736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10135375 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10145351 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 562036736 _runtime‚ 947.696804587 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚167011448268800 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚36927418845184 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165014332264448 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8640394 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10145351 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8640394 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4285959 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115578368 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚167011806646784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚87 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚167011806646784 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96966144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2355825 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1165660142080 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8640392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 562036736 trainer/global_step‚1099 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚165860106878976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166163925067479 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165860089839616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165014349303808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2355541 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24404557824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1437 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1165757108224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1149592803031 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165860089839616 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165860106878976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1437 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚38093078987264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7843636691871116e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚167011448268800 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4286246 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0W »wr0Å+ _wandbruntime‚948  _runtime‚948,oÞ&ÆØM trainer/global_step‚1099 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10144427 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36960847603712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1152712401920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1166685826048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1150944406751 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1932166 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1932163 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚88 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10154417 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167160060310528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167159701925376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8648026 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 _runtime‚ 948.323120253 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165160842936320 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8648024 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚166311446793073 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18802443 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10144427 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167160060310528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165160859975680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1150603856753 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166007347908608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2357612 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1152371056128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843636698168483e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1152371056128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4290044 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚268 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10144427 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚167160060310528 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 562036736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1437 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24404557824 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24404557824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚268 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12100 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165160859975680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1150603856753 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166007347908608 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2357612 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88570368 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1152371056128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843636698168483e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1152371056128 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4290044 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚38127533429760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10144427 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36960847603712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1152712401920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1166685826048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1150944406751 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18792451 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1932166 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1932163 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚88 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10154417 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚167160060310528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚167159701925376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24404557824 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8648026 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4289775 _runtime‚ 948.323120253 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10154417 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1437X̱˜0È+ _wandbruntime‚948  _runtime‚9488‚=ÏÉý 0loss/FRt2/assignment_loss‚1.9856157893383946 2loss/SRqqt1/assignment_loss‚1.2570193227056576 2loss/SRqqt2/assignment_loss‚0.9872442719984947 1loss/FBt1/assignment_loss‚0.10438618306756284 /loss/FRt1/detection_loss‚0.6757821605982884 /loss/FRt2/detection_loss‚1.4290582944475183 1loss/SRqqt2/detection_loss‚0.7724058323992535 0loss/FRt1/assignment_loss‚3.0370862939010443 _step‚89 /loss/FBt2/detection_loss‚1.2673010043799877 $loss/total_loss‚13.7688000727324 epoch‚2 1loss/FBt2/assignment_loss‚0.20569547167694205 $ _timestamp‚1.784363669817975e+09 1loss/SRqqt1/detection_loss‚1.4890428567062253 trainer/global_step‚1099 _runtime‚ 948.323725946 /loss/FBt1/detection_loss‚0.5581625915130296=’>ŒÊ 1loss/SRqqt1/detection_loss‚1.4890428567062253 trainer/global_step‚1099 _runtime‚ 948.323725946 /loss/FBt1/detection_loss‚0.5581625915130296 0loss/FRt2/assignment_loss‚1.9856157893383946 2loss/SRqqt1/assignment_loss‚1.2570193227056576 2loss/SRqqt2/assignment_loss‚0.9872442719984947 1loss/FBt1/assignment_loss‚0.10438618306756284 /loss/FRt1/detection_loss‚0.6757821605982884 /loss/FRt2/detection_loss‚1.4290582944475183 1loss/SRqqt2/detection_loss‚0.7724058323992535 0loss/FRt1/assignment_loss‚3.0370862939010443 _step‚89 /loss/FBt2/detection_loss‚1.2673010043799877 $loss/total_loss‚13.7688000727324 epoch‚2 1loss/FBt2/assignment_loss‚0.20569547167694205 $ _timestamp‚1.784363669817975e+09YÏxGŸË‚(:” ¡õìÒ¿Ùþí gpu.0.gpu‚96.0 gpu.0.memory‚34, gpu.0.memoryAllocated‚29.045486450195312* gpu.0.memoryAllocatedBytes‚ 24949882880 gpu.0.temp‚35.0 gpu.0.powerWatts‚142.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.586399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì.z«Ì‚(:  ¡õìÒꪲï proc.cpu.threads‚17 network.recv‚299301 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 20.703125 network.sent‚ 1.601608e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 1027.84375 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2728.21875 disk.dm-1.in‚0 disk.dm-1.out‚ 20.703125( proc.memory.availableMB‚ 251.38671875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚2.5625 disk.nvme0c0n1.in‚ 1027.84375% disk.nvme0c0n1.out‚2904.498046875) disk./.usagePercent‚39.57103157396217 disk.sda2.in‚0# memory_percent‚98.4656572341919) proc.memory.percent‚16.65172576904297 cpu‚25.841760847422137$ disk./.usageGB‚589.6369667053223 disk.sda3.out‚ 20.703125# disk.nvme0n1.out‚2904.498046875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1027.84375 disk.dm-0.out‚2904.498046875àO¥{’Í‚(:‡ °õìÒ‹ò¯í gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷¯˱΂(:¦ °õìÒ™íÙï disk.sda2.out‚0 network.recv‚303716 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.65172576904297 cpu‚25.839616987827313 disk.rbd1.out‚2.640625$ memory_percent‚98.46820831298828 disk.dm-0.in‚ 1027.84375$ disk./.usageGB‚589.6368827819824 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 1.664454e+06 disk.dm-1.in‚0% disk.nvme0c0n1.out‚2907.537109375 disk.sda.out‚ 20.91796875 disk.nvme0c0n1.in‚ 1027.84375 disk.sda1.out‚0 disk.dm-0.out‚2907.537109375) disk./.usagePercent‚39.57102594179642 disk.drbd2165.in‚0# disk.nvme0n1.out‚2907.537109375 proc.memory.rssMB‚ 2728.21875 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 1027.84375 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 20.91796875 disk.dm-1.out‚ 20.91796875% proc.memory.availableMB‚ 250.96875Šˆ,"0Ï+ _wandbruntime‚988  _runtime‚988A^ÆÂм % lr-AdamW/pg1‚0.000579928071421845 % lr-AdamW/pg2‚0.000579928071421845 trainer/global_step‚1149 $ _timestamp‚1.784363710266993e+09 _runtime‚ 988.772932675 _step‚90J½ÆÑÀ % lr-AdamW/pg2‚0.000579928071421845 trainer/global_step‚1149 $ _timestamp‚1.784363710266993e+09 _runtime‚ 988.772932675 _step‚90 % lr-AdamW/pg1‚0.000579928071421845Z…VÏé0Ò+ _wandbruntime‚988  _runtime‚988F~»ÊÓÎO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1217098020864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173222141321216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4476820 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10589827 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 _step‚91 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2459407 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1202395269632 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10599803 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚ 988.774240394 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1202395269632 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2017671 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1202053931008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174424553630208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1200549839825–ÁYØ'ÔÒO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1202053931008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9021992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2459152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173222158360576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38598856771584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172339865858048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174424195252224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2017671 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1202053931008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174424553630208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚174424553630208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10589827 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1200549839825 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1217098020864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1217007346176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173222141321216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4476820 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10589827 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 _step‚91 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2459407 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1202395269632 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10599803 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12100 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚ 988.774240394 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1202395269632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173369382350848 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1203407903744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1203066557952 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9029624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10608869 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚1149 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2019413 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1203066557952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784363710897052e+09 _step‚92 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174572448908800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2461481 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173687954462939 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173687596873581 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174572807293952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10608869 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2461208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173369382350848 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173369382350848 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9029626 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1203407903744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554757632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173369399390208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1203066557952 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9990 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19628503 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 \JDeviceStatsŽÏâ"óMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1201560893659 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9029624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚19628503 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341345792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10608869 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4480897 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38632285530112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 trainer/global_step‚1149 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚172486376529920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2019413 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1203066557952 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784363710897052e+09 _step‚92 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174572448908800 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10598879 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10598879 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2461481 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173687954462939 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1201220343661 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4480621 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚172486393569280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173687596873581 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚174572807293952 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10608869 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587\+§òf0Ø+ _wandbruntime‚989  _runtime‚989ê$%Ùû .loss/FBt2/detection_loss‚1.100115206819482 /loss/FRt2/assignment_loss‚1.922678646147858 2loss/SRqqt1/assignment_loss‚1.2077837746823206 2loss/FBt2/assignment_loss‚0.052513750371822536 /loss/FRt2/detection_loss‚1.4796683410070686 1loss/SRqqt2/detection_loss‚0.8211369089689934 epoch‚2 trainer/global_step‚1149 _runtime‚ 989.404835888 0loss/FRt1/assignment_loss‚2.9586461292986836 2loss/SRqqt2/assignment_loss‚0.9302403380434009 /loss/SRqqt1/detection_loss‚1.32129685273394 % _timestamp‚1.7843637108991551e+09 _step‚93 0loss/FBt1/assignment_loss‚0.1151151948181808 /loss/FRt1/detection_loss‚0.6490686355016315 /loss/FBt1/detection_loss‚0.5018481458945997 %loss/total_loss‚13.06011192428798”kÃÛÚÿ epoch‚2 trainer/global_step‚1149 _runtime‚ 989.404835888 0loss/FRt1/assignment_loss‚2.9586461292986836 2loss/SRqqt2/assignment_loss‚0.9302403380434009 /loss/SRqqt1/detection_loss‚1.32129685273394 % _timestamp‚1.7843637108991551e+09 _step‚93 0loss/FBt1/assignment_loss‚0.1151151948181808 /loss/FRt1/detection_loss‚0.6490686355016315 /loss/FBt1/detection_loss‚0.5018481458945997 %loss/total_loss‚13.06011192428798 .loss/FBt2/detection_loss‚1.100115206819482 /loss/FRt2/assignment_loss‚1.922678646147858 2loss/SRqqt1/assignment_loss‚1.2077837746823206 2loss/FBt2/assignment_loss‚0.052513750371822536 /loss/FRt2/detection_loss‚1.4796683410070686 1loss/SRqqt2/detection_loss‚0.8211369089689934]Š„rÛ‚(:’ ¿õìÒÃëêì gpu.0.gpu‚86.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ÂÜ‚(:· ¿õìÒÑØ”ï disk.drbd2165.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚2912.32666015625 disk.sda.in‚0 disk.sda1.out‚0 network.sent‚ 1.669371e+06 disk.rbd1.in‚0 proc.cpu.threads‚17' disk.nvme0c0n1.out‚2912.32666015625 network.recv‚307018* disk./.usagePercent‚39.571055638670394! disk.nvme0n1.in‚ 1106.58984375" disk.dm-0.out‚2912.32666015625 disk.rbd1.out‚2.640625 proc.memory.rssMB‚ 2728.21875 disk.dm-1.out‚ 21.49609375 disk.drbd2165.out‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 1106.58984375 disk.sda.out‚ 21.49609375 cpu‚25.837828580473253 disk.dm-0.in‚ 1106.58984375 disk.sda2.out‚0 disk.sda1.in‚0$ disk./.usageGB‚589.6373252868652$ memory_percent‚98.46963882446289& proc.memory.availableMB‚ 250.734375 disk.sda3.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0) proc.memory.percent‚16.65172576904297 disk.sda3.out‚ 21.496093756Ä5דÝ‚(:ˆ ÎõìÒ磌ï gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Å]αÞ‚(:¦ ÎõìÒÉ™ð& disk.nvme0c0n1.out‚2915.8115234375$ memory_percent‚98.46861362457275 disk.dm-1.out‚21.875 disk.drbd2165.out‚0! disk.dm-0.out‚2915.8115234375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚2.703125 cpu‚25.835766905373703 disk.rbd0.out‚0$ disk.nvme0n1.out‚2915.8115234375 network.recv‚311362$ disk./.usageGB‚589.6367874145508) proc.memory.percent‚16.65172576904297 disk.rbd0.in‚0 proc.memory.rssMB‚ 2728.21875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 1106.58984375( proc.memory.availableMB‚ 250.90234375 disk.sda.out‚21.875 disk.sda1.out‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 1106.58984375) disk./.usagePercent‚39.57101954160807 disk.sda3.out‚21.875 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda.in‚0 network.sent‚ 1.734412e+06# disk.nvme0c0n1.in‚ 1106.58984375{.8,“ß‚(:ˆ ÝõìÒÓî‡í gpu.0.gpu‚41.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>è¿à‚(:´ ÝõìÒ‰‚ï disk.drbd2165.in‚0" disk.dm-0.out‚2920.24365234375 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.percent‚16.65172576904297 cpu‚25.834117685261393 disk.sda3.in‚0' disk.nvme0c0n1.out‚2920.24365234375 network.recv‚314664 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚98.46725463867188 network.sent‚ 1.737376e+06 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda.out‚ 22.03515625 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1106.58984375% disk.nvme0n1.out‚2920.24365234375 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 1106.58984375 disk.sda.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚39.571017749555324 disk.sda3.out‚ 22.03515625# proc.memory.availableMB‚251.125 proc.memory.rssMB‚ 2728.21875 disk.dm-0.in‚ 1106.58984375 disk.rbd1.in‚0 disk.rbd1.out‚2.703125$ disk./.usageGB‚589.6367607116699 disk.dm-1.out‚ 22.03515625´“ì2á- _wandbruntime‚1029  _runtime‚1029/žÆâÀ & lr-AdamW/pg1‚0.0005796394738151335 & lr-AdamW/pg2‚0.0005796394738151335 trainer/global_step‚1199 % _timestamp‚1.7843637513072782e+09 _runtime‚1029.813252204 _step‚94ký ÊãÄ & lr-AdamW/pg1‚0.0005796394738151335 & lr-AdamW/pg2‚0.0005796394738151335 trainer/global_step‚1199 % _timestamp‚1.7843637513072782e+09 _runtime‚1029.813252204 _step‚94^-832ä- _wandbruntime‚1029  _runtime‚1029óÌáÿÔ'åÎO IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253090771456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11044279 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2104918 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180916582817741 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20447871 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2562791 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181836942235648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252749432832 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41538646192640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180584192802816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1251166326733 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41538553682944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252749432832 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4667451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179665416491008 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9403592 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509696 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9403594 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180584209842176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40270294697984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246G+àØ'æÒO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341338624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268332882432 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179665399451648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚92509696 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9403594 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180584209842176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40270294697984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702948864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589246 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181836942235648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1253090771456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40270313310208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11044279 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2104918 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚180916582817741 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20447871 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2562791 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358377984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181836942235648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1252749432832 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚41538646192640 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚180916225228495 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358377984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11044279 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056587 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180584192802816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341338624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1251166326733 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702948864 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554757632 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚41538553682944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549886 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12100 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1252749432832 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4667451 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179665416491008 trainer/global_step‚1199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9403592 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169_f½~2ç- _wandbruntime‚1030  _runtime‚1030ÁòÄ$’èØM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11053331 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251836830569 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2106666 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 _runtime‚1030.446024987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254103405568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181064104543447 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180731450871808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 % _timestamp‚1.7843637519396434e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4671449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2564783 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚1199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663†~Ûâ&éÜM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4671195 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11053331 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181063746954089 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9411226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251836830569 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056587 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702948864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2106666 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1252177380567 _runtime‚1030.446024987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181985554277376 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254103405568 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9990 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1253762059776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181064104543447 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589358 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303742068736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚20464555 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180731450871808 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303723456512 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340549998 % _timestamp‚1.7843637519396434e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20474547 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554757632 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180731433832448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269284976128 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967979 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702948864 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4671449 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2564783 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2564532 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 trainer/global_step‚1199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚41573008432640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2106663 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341345792 B&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6667443513870239 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9995090961456299 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚1199 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936547875404358 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13148896396160126 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5412216782569885 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6252834796905518 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41435185074806213 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5116707682609558 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5078246593475342 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4976274371147156 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898081421852112 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08032912015914917 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.051792047917842865 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17424637079238892 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00735930260270834 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06108815222978592 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06077444553375244 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.49988818168640137 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.009224173612892628 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4732320010662079 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5750566720962524 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5741017460823059 +jet/accuracy_2_of_2‚0.07510173320770264 % _timestamp‚1.7843637693423133e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7validation_average_jet_accuracy‚0.27071958780288696 *loss/val_total_loss‚13.367801814851262 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5883203148841858 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.0746542438864708 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 /particle/accuracy_1_of_2‚0.9998962879180908 /particle/accuracy_2_of_2‚0.9998962879180908 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6852985620498657 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9886727333068848 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.047655656933784485 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5239426493644714 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06377610564231873 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3097127377986908 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.060833755880594254 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/eve(þÐnt_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196357727050781 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5404772758483887 particle/accuracy_1_of_4‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4637686014175415 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6077591776847839 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.46958333253860474 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )jet/accuracy_1_of_4‚0.793073296546936 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 _step‚98 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6168492436408997 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16456294059753418 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09962716698646545 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4977269768714905 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5854526162147522 jet/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996226966381073 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5402025580406189 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8170223236083984 jet/accuracy_4_of_5‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06383727490901947 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5197262167930603 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.23348695039749146 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009108553640544415 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12349209189414978 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.28240740299224854 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5872116684913635 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.5620216131210327 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 *jet/accuracy_2_of_3‚0.3047489821910858 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5017287731170654 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865644574165344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9947282075881958 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6060271263122559 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8789771795272827 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07345060259103775 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0724082961678505 *jet/accuracy_1_of_2‚0.6518173217773438 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.46777334809303284 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7603617310523987 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5624748468399048 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.22796235978603363 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06517116725444794 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07176906615495682 epoch‚2 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966605305671692 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6777966618537903 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.989861786365509 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.549744188785553 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5274351835250854 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951285123825073 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.11100786179304123 ,jet/accuracy_3_of_4‚0.038126252591609955 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.30000874400138855 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6991118788719177 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5267051458358765 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375èTbê=ñä{ 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7603617310523987 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5624748468399048 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.22796235978603363 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.06517116725444794 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.07176906615495682 epoch‚2 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966605305671692 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6777966618537903 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.989861786365509 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.549744188785553 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5274351835250854 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951285123825073 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.11100786179304123 ,jet/accuracy_3_of_4‚0.038126252591609955 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.30000874400138855 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6991118788719177 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5267051458358765 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966605305671692 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07348193973302841 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08445018529891968 *particle/sensitivity‚0.546707272529602 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5530981421470642 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9865193963050842 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5475953221321106 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4619634747505188 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0589858815073967 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882463812828064 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.08148350566625595 /particle/accuracy_3_of_4‚0.9970238208770752 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5846371054649353 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5274407267570496 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9773454070091248 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9905997514724731 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.030794110149145126 _runtime‚1047.848963509 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5524756908416748 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0601179264485836 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.06686273962259293 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 6 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.996457576751709 *jet/accuracy_1_of_3‚0.7820013165473938 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19800803065299988 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5402025580406189 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6418955326080322 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004583441652357578 particle/accuracy_2_of_3‚1 .particle/accuracy_4_of_4‚0.947913646697998 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07261005789041519 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2890625 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.060662951320409775 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6685185432434082 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5689710378646851 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.044276706874370575 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940147399902344 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10144682973623276 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5585759878158569 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4220598638057709 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9926154017448425 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992692232131958 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08628486096858978 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9939857721328735 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 +jet/accuracy_2_of_4‚0.42602354288101196 ,jet/accuracy_4_of_4‚0.007130124606192112 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.567450225353241 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0631026029586792 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05885075405240059 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5116707682609558 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05630654841661453 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7759310603141785 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.09493373334407806 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.0724082961678505 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8212487101554871 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.08247517049312592 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05885075405240059 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.22193747758865356 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 +particle/specificity‚0.8909508585929871 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970272779464722 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9952360987663269 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06357607245445251 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4255782961845398 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5854706764221191 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6667443513870239 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_3_of_3‚0.9995090961456299 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚1199 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936547875404358 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13148896396160126 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5412216782569885 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6252834796905518 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41435185074806213 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5116707682609558 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5078246593475342 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4976274371147156 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898081421852112 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08032912015914917 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.051792047917842865 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17424637079238892 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.00735930260270834 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06108815222978592 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06077444553375244 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.49988818168640137 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.009224173612892628 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4732320010662079 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5750566720962524 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5741017460823059 +jet/accuracy_2_of_2‚0.07510173320770264 % _timestamp‚1.7843637693423133e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7validation_average_jet_accuracy‚0.27071958780288696 *loss/val_total_loss‚13.367801814851262 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5883203148841858 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.0746542438864708 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 /particle/accuracy_1_of_2‚0.9998962879180908 /particle/accuracy_2_of_2‚0.9998962879180908 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791865348816 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6852985620498657 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9886727333068848 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.047655656933784485 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5239426493644714 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06377610564231873 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3097127377986908 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.060833755880594254 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196357727050781 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5404772758483887 particle/accuracy_1_of_4‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4637686014175415 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6077591776847839 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.46958333253860474 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )jet/accuracy_1_of_4‚0.793073296546936 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2291666716337204 _step‚98 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6168492436408997 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16456294059753418 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.09962716698646545 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4977269768714905 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_1‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5854526162147522 jet/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996226966381073 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5402025580406189 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8170223236083984 jet/accuracy_4_of_5‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06383727490901947 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5197262167930603 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.23348695039749146 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009108553640544415 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.12349209189414978 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.28240740299224854 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5872116684913635 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.5620216131210327 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 *jet/accuracy_2_of_3‚0.3047489821910858 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5017287731170654 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9865644574165344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9947282075881958 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6060271263122559 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8789771795272827 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9980952143669128 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.07345060259103775 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.0724082961678505 *jet/accuracy_1_of_2‚0.6518173217773438 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.46777334809303284bõèÍGò‚(:… ûõìÒî–¯î gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚30.0 gpu.0.powerWatts‚91.507' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚18.3014 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÊíô»ó‚(:° ûõìÒ¤  ï" disk.dm-0.out‚3177.40966796875 disk.rbd1.out‚2.78125" disk.nvme0c0n1.in‚ 1170.6484375' disk.nvme0c0n1.out‚3177.40966796875 disk.sda1.in‚0 disk.dm-0.in‚ 1170.6484375$ disk./.usageGB‚589.8773536682129 disk.sda.out‚ 22.578125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 22.578125 disk.rbd0.out‚0 network.sent‚ 1.805349e+06 disk.sda2.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 1170.6484375) proc.memory.percent‚16.42603874206543 cpu‚25.894096139588527 proc.cpu.threads‚17( proc.memory.availableMB‚ 495.16015625 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 22.578125" proc.memory.rssMB‚ 2691.2421875% disk.nvme0n1.out‚3177.40966796875 network.recv‚322686 disk.dm-1.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚39.58716414473986 disk.rbd1.in‚0$ memory_percent‚96.97778224945068ÜgAô‚(:’ ŠöìÒºÃøì gpu.0.gpu‚50.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£'ŽlÁõ‚(:¶ ŠöìÒ‡±±ï$ disk./.usageGB‚589.8775978088379 disk.sda3.out‚ 22.90234375 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 22.90234375% disk.nvme0n1.out‚3182.24462890625) proc.memory.percent‚16.42482280731201 cpu‚25.891417863073695 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚3182.24462890625' proc.memory.availableMB‚ 494.8671875 network.recv‚326876* disk./.usagePercent‚39.587180529222046 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1170.6484375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 22.90234375" disk.nvme0c0n1.in‚ 1170.6484375 disk.drbd2165.in‚0 disk.rbd1.out‚2.84375 disk.nvme0n1.in‚ 1170.6484375" disk.dm-0.out‚3182.24462890625 disk.sda2.in‚0 proc.cpu.threads‚17$ memory_percent‚96.97957038879395# proc.memory.rssMB‚ 2691.04296875 network.sent‚ 1.86273e+06êÙožö‚(:“ ™öìÒÑÐûì gpu.0.gpu‚88.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚249.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.830200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý{ÃC¼÷‚(:± ™öìÒÓãî disk.sda3.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 1170.6484375 disk.sda.out‚ 23.1484375# proc.memory.rssMB‚ 2691.04296875) proc.memory.percent‚16.42482280731201 disk.rbd0.in‚0 network.sent‚ 1.867647e+06 disk.sda2.in‚0 disk.rbd1.out‚2.84375' disk.nvme0c0n1.out‚3433.89599609375 disk.sda1.out‚0& proc.memory.availableMB‚ 494.859375 disk.dm-0.in‚ 1170.6484375 disk.rbd1.in‚0 disk.sda1.in‚0$ disk./.usageGB‚589.8766136169434 disk.rbd0.out‚0 disk.nvme0n1.in‚ 1170.6484375 proc.cpu.threads‚17 network.recv‚330258) disk./.usagePercent‚39.58711447927822% disk.nvme0n1.out‚3433.89599609375 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 23.1484375 disk.drbd2165.out‚0 disk.sda3.out‚ 23.1484375 disk.dm-1.in‚0" disk.dm-0.out‚3433.89599609375$ memory_percent‚96.97961807250977 cpu‚25.88862260948074-ëf2ø- _wandbruntime‚1090  _runtime‚1090¬'zÆùÀ _runtime‚1090.270753523 _step‚99 & lr-AdamW/pg1‚0.0005793323381637578 & lr-AdamW/pg2‚0.0005793323381637578 trainer/global_step‚1249 % _timestamp‚1.7843638117646484e+09möÑÊúÄ % _timestamp‚1.7843638117646484e+09 _runtime‚1090.270753523 _step‚99 & lr-AdamW/pg1‚0.0005793323381637578 & lr-AdamW/pg2‚0.0005793323381637578 trainer/global_step‚1249c Zr,2û- _wandbruntime‚1090  _runtime‚10909Ÿ!‹³üÏO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21474914 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚189875827974144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/@møb"active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188899613298688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1321178141696 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905860 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2194688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905858 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189875827974144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1321519480832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1337517365760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319161988984 % _timestamp‚1.7843638117658238e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41957644724224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚189875810934784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190219115837562 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190218758248312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0«/Ù'ýÓO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41957644724224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚189875810934784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚190219115837562 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚190218758248312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21474914 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚191196989076480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚189875827974144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188899613298688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1321178141696 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9905860 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2194688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9905858 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚189875827974144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1321519480832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1337517365760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319161988984 % _timestamp‚1.7843638117658238e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0d—ÃÙó2þ- _wandbruntime‚1090  _runtime‚1090–®ãà&ÿÚM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11578108 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338456782848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320173042706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11578108 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189046123970560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190023069003776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2196433 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41991073482752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1322190768640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1090.902548499 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190023051964416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4894539 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1322532114944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1322532114944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190023051964416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438…ëëxÐ!€ÞM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11578108 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9913492 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1338456782848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320173042706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11578108 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4894258 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189046123970560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190023069003776 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2196433 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11588099 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41991073482752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚191345601118720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1322190768640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1090.902548499 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190023051964416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚191345601118720 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1249 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4894539 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.fre=Ù^Ëed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1322532114944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1322532114944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190023051964416 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2698103 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚278e6b³2- _wandbruntime‚1090  _runtime‚1090…õÜ‚ý /loss/FRt1/assignment_loss‚2.918974280295466 1loss/FBt2/assignment_loss‚0.09974570978724492 _step‚102 0loss/FRt2/assignment_loss‚1.8974896690139917 1loss/SRqqt2/detection_loss‚0.8017298113961199 /loss/FBt1/detection_loss‚0.5892984914783113 &loss/total_loss‚12.753692362675295 trainer/global_step‚1249 $ _timestamp‚1.784363812398155e+09 1loss/SRqqt1/assignment_loss‚1.267679669292627 1loss/SRqqt2/assignment_loss‚0.902732517028231 1loss/FBt1/assignment_loss‚0.06912129111892908 /loss/FRt1/detection_loss‚0.6518852857349459 /loss/FRt2/detection_loss‚1.4190946157757784 _runtime‚1090.903843594 1loss/SRqqt1/detection_loss‚1.1962634817556996 .loss/FBt2/detection_loss‚0.939677539997949 epoch‚3¸(Zuƒ /loss/FRt1/assignment_loss‚2.918974280295466 1loss/FBt2/assignment_loss‚0.09974570978724492 _step‚102 0loss/FRt2/assignment_loss‚1.8974896690139917 1loss/SRqqt2/detection_loss‚0.8017298113961199 /loss/FBt1/detection_loss‚0.5892984914783113 &loss/total_loss‚12.753692362675295 trainer/global_step‚1249 $ _timestamp‚1.784363812398155e+09 1loss/SRqqt1/assignment_loss‚1.267679669292627 1loss/SRqqt2/assignment_loss‚0.902732517028231 1loss/FBt1/assignment_loss‚0.06912129111892908 /loss/FRt1/detection_loss‚0.6518852857349459 /loss/FRt2/detection_loss‚1.4190946157757784 _runtime‚1090.903843594 1loss/SRqqt1/detection_loss‚1.1962634817556996 .loss/FBt2/detection_loss‚0.939677539997949 epoch‚3f“,_“„‚(:ˆ ¨öìÒÇûÉí gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÎDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370119025782 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚1299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚197237862416384 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2799156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197237879455744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚197237862416384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196225146892288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12023508H«šÞ'ØO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2799156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚197237879455744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚197237862416384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12033485 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198609736059904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196225146892288 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12023508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43629082650624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚197237879455744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1369778475892 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚198610094438400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1371873643520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1388744171008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7843638526177876e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198609736059904 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2281941 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1370119025782 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚1299 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚197237862416384 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0hþ†Ö›2- _wandbruntime‚1131  _runtime‚1131M©1ƒn‘ÝM NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5084884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 1131.75414926 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1373227616768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2283683 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12042551 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10295090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196371640524800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚1299 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12032560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843638532480044e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10295092 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12032560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2800946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12042551 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197742430054414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197385103446016 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198758348102144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1370789529614 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197385120485376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22327650 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1389675437056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196371657564160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10295090 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43662530021376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371130079616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976©çtÆç&’áM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚105 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198758348102144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1372886270464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2801198 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1371130079616 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5084884 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚ 1131.75414926 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5084629 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1373227616768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2283683 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43662511409152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12042551 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10295090 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196371640524800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚1299 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12032560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843638532480044e+09 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10295092 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12032560 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2800946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12042551 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197742430054414 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚197385103446016 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198758348102144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1370789529614 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚197385120485376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22327650 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45052291931136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1389675437056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196371657564160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10295090 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22337643 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43662530021376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 1172.15452511 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12477960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12487937 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10669058 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12477960 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1420735512690 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204971058409328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1420394962800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1422569145344 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5271482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206022483043328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439969710592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1422569145344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203550680485888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚ 1172.15452511 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12477960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12487937 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10669058 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12477960 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5271207 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2902019 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206022483043328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1420735512690 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚204599930937344 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204971058409328 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1420394962800 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1422569145344 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5271482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚204599930937344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206022483043328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439969710592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23147018 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1422569145344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203550680485888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5275293 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843638942756703e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2904357 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204747154927616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206170736699904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204747171966976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12497003 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676690 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12487012 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45333949335552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚1349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203697174118400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10676690 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚109 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2904096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204747171966976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203697191157760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2370936 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888cgç&¤áM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚204747171966976 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12497003 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10676690 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1421406016522 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1423581772288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12487012 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46774960418816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45333949335552 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚1349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203697174118400 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10676690 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚109 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2904096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚204747171966976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203697191157760 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2370936 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10676692 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5275293 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843638942756703e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5275029 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2904357 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12497003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12487012 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚204747154927616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206170736699904 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088mk ùv2¥- _wandbruntime‚1172  _runtime‚1172*ä¦ý /loss/FRt1/assignment_loss‚3.008893659828119 0loss/SRqqt1/detection_loss‚1.350105007263747 /loss/FBt2/detection_loss‚1.7979902688946043 trainer/global_step‚1349 .loss/FRt2/detection_loss‚1.539285296511064 /loss/FBt1/detection_loss‚2.5188185265572516 /loss/FRt2/assignment_loss‚1.997969993209654 0loss/FBt2/assignment_loss‚0.1423898417971056 /loss/FRt1/detection_loss‚0.6483732883841766 &loss/total_loss‚15.977949987054384 _step‚110 2loss/SRqqt1/assignment_loss‚1.0566677938163023 2loss/SRqqt2/assignment_loss‚0.9185914409350535 1loss/FBt1/assignment_loss‚0.11305788112882423 1loss/SRqqt2/detection_loss‚0.8858069887284814 epoch‚3 % _timestamp‚1.7843638942773292e+09 _runtime‚1172.783008236ŽD˜Ê§ .loss/FRt2/detection_loss‚1.539285296511064 /loss/FBt1/detection_loss‚2.5188185265572516 /loss/FRt2/assignment_loss‚1.997969993209654 0loss/FBt2/assignment_loss‚0.1423898417971056 /loss/FRt1/detection_loss‚0.6483732883841766 &loss/total_loss‚15.977949987054384 _step‚110 2loss/SRqqt1/assignment_loss‚1.0566677938163023 2loss/SRqqt2/assignment_loss‚0.9185914409350535 1loss/FBt1/assignment_loss‚0.11305788112882423 1loss/SRqqt2/detection_loss‚0.8858069887284814 epoch‚3 % _timestamp‚1.7843638942773292e+09 _runtime‚1172.783008236 /loss/FRt1/assignment_loss‚3.008893659828119 0loss/SRqqt1/detection_loss‚1.350105007263747 /loss/FBt2/detection_loss‚1.7979902688946043 trainer/global_step‚1349n9‚+Å“¨‚(:ˆ ‚÷ìÒëðôî gpu.0.gpu‚86.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÈ&³©‚(:¨ ‚÷ìÒúñ network.sent‚ 2.030359e+06' disk.nvme0c0n1.out‚3461.96435546875 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 494.6015625 disk.rbd1.in‚0% disk.nvme0n1.out‚3461.96435546875 disk.sda.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚355689) disk./.usagePercent‚39.58713649592616 disk.nvme0n1.in‚ 1319.1875 disk.sda2.out‚0# proc.memory.rssMB‚ 2691.04296875 disk.sda3.out‚ 25.5859375 disk.dm-0.in‚ 1319.1875 disk.sda2.in‚0$ disk./.usageGB‚589.8769416809082 disk.rbd1.out‚3.125 disk.nvme0c0n1.in‚ 1319.1875 disk.sda.out‚ 25.5859375 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.percent‚16.42482280731201 disk.drbd2165.in‚0$ memory_percent‚96.98119163513184 disk.sda1.out‚0 disk.dm-1.out‚ 25.5859375" disk.dm-0.out‚3461.96435546875 cpu‚25.871532021725024¦óÌ‚žª‚(:“ ‘÷ìÒ“”õî gpu.0.gpu‚83.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï0ã«‚(:¬ ‘÷ìÒ³þ™ñ disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚96.97988033294678 disk.sda3.out‚ 25.77734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 25.77734375" disk.dm-0.out‚µÜÍÔ4280.37548828125' disk.nvme0c0n1.out‚4280.37548828125 disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 1319.1875 disk.nvme0c0n1.in‚ 1319.1875$ disk./.usageGB‚590.6712532043457 disk.sda3.in‚0% disk.nvme0n1.out‚4280.37548828125 disk.rbd1.out‚3.125 disk.sda.out‚ 25.77734375# proc.memory.rssMB‚ 2691.04296875) disk./.usagePercent‚39.64044340873581) proc.memory.percent‚16.42482280731201 network.sent‚ 2.082298e+06 network.recv‚360309 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 494.81640625 cpu‚25.869831044542632 disk.drbd2165.in‚0 disk.dm-0.in‚ 1319.1875 disk.sda2.in‚0ïXÑ2¬- _wandbruntime‚1213  _runtime‚1213g™yÇ­Á trainer/global_step‚1399 % _timestamp‚1.7843639345153818e+09 _runtime‚1213.022833769 _step‚111 & lr-AdamW/pg1‚0.0005782999061547769 & lr-AdamW/pg2‚0.0005782999061547769g»Œ3ˮŠ& lr-AdamW/pg1‚0.0005782999061547769 & lr-AdamW/pg2‚0.0005782999061547769 trainer/global_step‚1399 % _timestamp‚1.7843639345153818e+09 _runtime‚1213.022833769 _step‚111o›/æ›2¯- _wandbruntime‚1213  _runtime‚1213âîUqØ'°ÒO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1473605986304 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213435588405248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1471351999598 trainer/global_step‚1399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5461962 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2456441 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11050660 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471011449708 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 # _timestamp‚1.78436393451618e+09 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48463031559168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1491073055744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212347208489836 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3005521 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48463140845568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46971958503424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11050658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12932412 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210876197040128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12942389 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1473605986304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12942389 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210876214079488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213435588405248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211961982418944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11050660 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1473264647168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _runtime‚1213.022935411 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154Ñ—¦AÜ'±ÖO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 # _timestamp‚1.78436393451618e+09 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48463031559168 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212347566079086 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1491073055744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1491163729920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212347208489836 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3005521 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48463140845568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46971958503424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11050658 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12932412 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210876197040128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12942389 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1473605986304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11050658 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12932412 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12942389 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210876214079488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213435588405248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211961982418944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211961982418944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11050660 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2456438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213435230026752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1473264647168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3005256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211961965379584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5461694 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _runtime‚1213.022935411 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46971977115648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1473605986304 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213435588405248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1471351999598 trainer/global_step‚1399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5461962 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2456441 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11050660 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471011449708 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169pd}2²- _wandbruntime‚1213  _runtime‚1213·˜è÷ã&³ÝM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212495087804792 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11058290 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12951455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2458186 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23999754 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1492095520256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213583842068992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1492181992960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48497482782208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213583842068992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5465755 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212494730215430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚1213.654700713 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212494730215430 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 _runtime‚1213.654700713 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1472022503430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12941464 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2458186 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2458183 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23999754 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1492095520256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213583842068992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1492181992960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212109223448576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11058292 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48497482782208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1474277274112 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3007302 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5465485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212109206409216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24009747 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1474277274112 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213583842068992qŸÆg2µ- _wandbruntime‚1213  _runtime‚1213É›•¶ÿ 2loss/SRqqt1/assignment_loss‚1.0760255669887764 1loss/SRqqt2/detection_loss‚0.7939451646008973 _step‚114 2loss/SRqqt2/assignment_loss‚0.7207277351570555 /loss/FRt1/detection_loss‚0.6799859996710376 1loss/SRqqt1/detection_loss‚1.1955586792613966 trainer/global_step‚1399 0loss/FRt2/assignment_loss‚1.9043010511004765 0loss/FBt2/assignment_loss‚0.1678782077328244 /loss/FRt2/detection_loss‚1.4590057365072537 /loss/FBt2/detection_loss‚1.1534445067086527 $loss/total_loss‚13.0365473724402 % _timestamp‚1.7843639351502967e+09 _runtime‚1213.655974699 0loss/FRt1/assignment_loss‚2.9901542908516054 1loss/FBt1/assignment_loss‚0.06928053987222857 /loss/FBt1/detection_loss‚0.8262398939879955 epoch‚3­ÒÍä ·ƒ _step‚114 2loss/SRqqt2/assignment_loss‚0.7207277351570555 /loss/FRt1/detection_loss‚0.6799859996710376 1loss/SRqqt1/detection_loss‚1.1955586792613966 trainer/global_step‚1399 0loss/FRt2/assignment_loss‚1.9043010511004765 0loss/FBt2/assignment_loss‚0.1678782077328244 /loss/FRt2/detection_loss‚1.4590057365072537 /loss/FBt2/detection_loss‚1.1534445067086527 $loss/total_loss‚13.0365473724402 % _timestamp‚1.7843639351502967e+09 _runtime‚1213.655974699 0loss/FRt1/assignment_loss‚2.9901542908516054 1loss/FBt1/assignment_loss‚0.06928053987222857 /loss/FBt1/detection_loss‚0.8262398939879955 epoch‚3 2loss/SRqqt1/assignment_loss‚1.0760255669887764 1loss/SRqqt2/detection_loss‚0.7939451646008973r&sñ¸‚(:’  ÷ìÒíÕ“ï gpu.0.gpu‚90.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+g᡹‚(:–  ÷ìÒï×­ð" disk.nvme0c0n1.out‚ 4283.265625 disk.dm-1.out‚26.03125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 1319.1875 disk.rbd1.out‚3.265625 disk.sda3.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2691.04296875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 1319.1875 cpu‚25.867611054742355 network.recv‚363642 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1319.1875 disk.dm-0.out‚ 4283.265625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.out‚26.03125 disk.drbd2165.in‚0$ memory_percent‚96.98150157928467( proc.memory.availableMB‚ 494.55078125 network.sent‚ 2.08522e+06$ disk./.usageGB‚590.6445274353027 disk.nvme0n1.out‚ 4283.265625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚26.03125) disk./.usagePercent‚39.63864981995093 disk.sda2.in‚0) proc.memory.percent‚16.42482280731201ÍCü¨“º‚(:ˆ ¯÷ìÒžÔ—í gpu.0.gpu‚83.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚260.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ä½J»»‚(:° ¯÷ìҊΘï disk.nvme0n1.in‚ 1403.65625% disk.nvme0n1.out‚4288.28564453125 network.sent‚ 2.148312e+06 network.recv‚368343 disk.dm-1.in‚0 disk.sda.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.59072981767676 disk.rbd0.in‚0" disk.dm-0.out‚4288.28564453125 disk.sda.out‚ 26.31640625 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 26.31640625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 495.0078125 disk.dm-0.in‚ 1403.65625 disk.rbd1.out‚3.265625$ memory_percent‚96.97871208190918# proc.memory.rssMB‚ 2691.04296875 cpu‚25.86525907395352$ disk./.usageGB‚589.9304847717285 disk.rbd1.in‚0) proc.memory.percent‚16.42482280731201 proc.cpu.threads‚17 disk.dm-1.out‚ 26.31640625 disk.nvme0c0n1.in‚ 1403.65625' disk.nvme0c0n1.out‚4288.28564453125 disk.sda2.out‚0 disk.drbd2165.in‚0/¾7¡’¼‚(:‡ ¾÷ìÒͨšï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚139.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.937 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"X¯ü²½‚(:§ ¾÷ìÒƒœáô disk.rbd0.out‚0 disk.nvme0n1.in‚ 1403.65625$ memory_percent‚96.98176383972168 network.sent‚ 2.155271e+06 disk.sda2.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda3.out‚26.65625 disk.dm-1.in‚0% disk.nvme0n1.out‚4292.33544921875' proc.memory.availableMB‚ 494.5078125 disk.rbd1.out‚3.328125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚ 1403.65625 disk.sda1.in‚0 disk.dm-0.in‚ 1403.65625# proc.memory.rssMB‚ 2691.04296875) proc.memory.percent‚16.42482280731201 cpu‚25.86385740848296 network.recv‚371625$ disk./.usageGB‚589.9305763244629' disk.nvme0c0n1.out‚4292.33544921875 disk.sda.in‚0 disk.sda.out‚26.65625" disk.dm-0.out‚4292.33544921875 disk.sda1.out‚0 disk.dm-1.out‚26.65625) disk./.usagePercent‚39.59073596185758z.´…2¾- _wandbruntime‚1254  _runtime‚1254±"¦/Ç¿Á & lr-AdamW/pg1‚0.0005779188274794103 & lr-AdamW/pg2‚0.0005779188274794103 trainer/global_step‚1449 % _timestamp‚1.7843639756390612e+09 _runtime‚1254.148669164 _step‚115K§·ñËÀÅ _runtime‚1254.148669164 _step‚115 & lr-AdamW/pg1‚0.0005779188274794103 & lr-AdamW/pg2‚0.0005779188274794103 trainer/global_step‚1449 % _timestamp‚1.7843639756390612e+09sn›äC2Á- _wandbruntime‚1254  _runtime‚12548'€IÚ'ÂÔO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚1254.149182053 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50185682272768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13386864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1523960148992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13396841 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1542285842944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219324033900544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50185795753472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13396841 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220848335388672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3108589 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219324016861184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1523960148992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48643396429824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219324033900544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5652004 trainer/global_step‚1449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1521627936616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3108316 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48643415042048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11432260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11432258 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13386864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843639756398191e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169‡‚Í'ðÃØO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13396841 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220848335388672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2543688 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3108589 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219324016861184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1523960148992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48643396429824 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚219324033900544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5652004 trainer/global_step‚1449 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1521627936616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220847977010176 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219723716159594 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3108316 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48643415042048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11432260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2543691 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11432258 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11432258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13386864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843639756398191e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5652280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218201730633728 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runt,H­ î ime‚1254.149182053 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚50185682272768 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13386864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1523960148992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13396841 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1542285842944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219324033900544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50185795753472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0tòOœ2Ä- _wandbruntime‚1254  _runtime‚1254ÄÈ ä&ÅÞM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1525314122240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220996589052416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218348241305600 trainer/global_step‚1449 _step‚117 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13405907 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48676843800576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439890 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219471274930176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1524972775936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11439890 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13395916 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1522979540340 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219871237885300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24835806‚ûè&ÆâM D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13395916 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2545436 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11439890 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13395916 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219870880295938 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1522979540340 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219871237885300 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24835806 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1525314122240 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220996589052416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218348241305600 trainer/global_step‚1449 _step‚117 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13405907 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5655767 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48676843800576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11439890 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219471274930176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1524972775936 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50314834356224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚1295.175113826 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1574655650816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228260723993600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3211793 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228261082372096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2630938 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1574996989952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1572584973414 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226686085382144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13851293 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228261082372096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226686085382144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226686068342784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225527281266688 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0<ÓàüÞ'ÕØO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1574655650816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13851293 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚227099866240102 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11813858 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25665153 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5842734 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50314852968448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1572244423524 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13841316 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51908378915840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1499 % _timestamp‚1.7843640166688545e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13841316 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225527264227328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226686068342784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1593428981760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚227099508650852 _step‚120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1574996989952 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51908263337984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11813860 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1572584973414 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226686085382144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13851293 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228260723993600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚228261082372096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚226686085382144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226686068342784 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225527281266688 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1593525947392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50314834356224 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚1295.175113826 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1574655650816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228260723993600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3211793 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228261082372096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3211528 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2630938 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1574996989952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1594543176192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1576009624064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1575668277760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50348281726976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1573596027248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1573255477246 trainer/global_step‚1499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1594444120576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225673774899200 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11821492 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226833326411776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1594543176192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5846282 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25681851 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1576009624064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1575668277760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11821490 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50348281726976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1573596027248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1573255477246 trainer/global_step‚1499 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1594444120576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51942707235328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚225673774899200 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11821492 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226833326411776 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976ycbó2Ù- _wandbruntime‚1295  _runtime‚1295SäeÚ 2loss/SRqqt1/assignment_loss‚1.0896014138321641 /loss/FRt1/detection_loss‚0.6768877128185473 % _timestamp‚1.7843640172989671e+09 1loss/FBt1/assignment_loss‚0.08659996862386049 1loss/SRqqt1/detection_loss‚1.3209739143789654 1loss/SRqqt2/detection_loss‚0.7654074674154518 /loss/FBt2/detection_loss‚1.0056150680673974 epoch‚3 _runtime‚1295.804802978 0loss/FRt2/assignment_loss‚1.8194536467583184 2loss/SRqqt2/assignment_loss‚0.6397497335952874 1loss/FBt2/assignment_loss‚0.11823162030634107 /loss/FRt2/detection_loss‚1.4758211430803254 trainer/global_step‚1499 _step‚122 /loss/FRt1/assignment_loss‚3.014043455809979 /loss/FBt1/detection_loss‚0.7866102852120007 &loss/total_loss‚12.798995429898639pí} Û… % _timestamp‚1.7843640172989671e+09 1loss/FBt1/assignment_loss‚0.08659996862386049 1loss/SRqqt1/detection_loss‚1.3209739143789654 1loss/SRqqt2/detection_loss‚0.7654074674154518 /loss/FBt2/detection_loss‚1.0056150680673974 epoch‚3 _runtime‚1295.804802978 0loss/FRt2/assignment_loss‚1.8194536467583184 2loss/SRqqt2/assignment_loss‚0.6397497335952874 1loss/FBt2/assignment_loss‚0.11823162030634107 /loss/FRt2/detection_loss‚1.4758211430803254 trainer/global_step‚1499 _step‚122 /loss/FRt1/assignment_loss‚3.014043455809979 /loss/FBt1/detection_loss‚0.7866102852120007 &loss/total_loss‚12.798995429898639 2loss/SRqqt1/assignment_loss‚1.0896014138321641 /loss/FRt1/detection_loss‚0.6768877128185473z`žÏN‘Ü‚(:† ú÷ìÒÆ®§ñ gpu.0.gpu‚36.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì*ÀÝ‚(:µ ú÷ìÒŒè‘ò disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda.in‚0) proc.memory.percent‚16.42482280731201 cpu‚25.856383460011354( proc.memory.availableMB‚ 248.22265625 proc.cpu.threads‚17* disk./.usagePercent‚39.588912164183654 disk.dm-1.out‚ 27.95703125 disk.rbd1.out‚3.46875& disk.nvme0c0n1.out‚4308.1259765625 disk.nvme0n1.in‚ 1486.0703125$ disk.nvme0n1.out‚4308.1259765625 disk.sda2.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2691.04296875 network.sent‚ 2.244116e+06 disk.dm-0.in‚ 1486.0703125! disk.dm-0.out‚4308.1259765625 disk.sda.out‚ 27.95703125 disk.sda3.in‚0 disk.sda3.out‚ 27.95703125 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚98.48496913909912 network.recv‚386840$ disk./.usageGB‚589.9034004211426" disk.nvme0c0n1.in‚ 1486.0703125 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0}ÔÈýÞ‚(:’ ‰øìÒ웆í gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.811' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.76219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ zBÃß‚(:¸ ‰øìÒÐùçï disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 28.22265625 disk.drbd2165.in‚0( proc.memory.availableMB‚ 247.98046875) proc.memory.percent‚16.42482280731201 network.recv‚391426 disk.rbd1.out‚3.46875# proc.memory.rssMB‚ 2691.04296875 cpu‚25.854360350499167 disk.dm-1.in‚0 disk.dm-0.in‚ 1486.0703125" disk.dm-0.out‚4312.60693359375 disk.sda2.in‚0 disk.rbd1.in‚0$ disk./.usageGB‚589.9034080505371% disk.nvme0n1.out‚4312.60693359375 disk.dm-1.out‚ 28.22265625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 1486.0703125* disk./.usagePercent‚39.588912676198724 disk.sda.in‚0 disk.sda.out‚ 28.22265625 disk.sda3.in‚0 disk.rbd0.in‚0$ memory_percent‚98.48644733428955 network.sent‚ 2.293966e+06 disk.sda2.out‚0' disk.nvme0c0n1.out‚4312.60693359375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0n1.in‚ 1486.0703125«Çพà‚(:“ ˜øìÒ˜š‰ð gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚265.314' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.062799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 4ÆPµá‚(:ª ˜øìÒ’›ñ disk.rbd0.out‚0 disk.sda2.out‚0$ memory_percent‚98.48344326019287 proc.cpu.threads‚17 disk.dm-1.out‚ 28.46484375 disk.drbd2165.in‚0( disk./.usagePercent‚39.5889259885905 disk.rbd0.in‚0 disk.dm-0.out‚ 4316.9609375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 1486.0703125 disk.dm-0.in‚ 1486.0703125 disk.sda2.in‚0 disk.rbd1.out‚3.546875 disk.sda.in‚0" proc.memory.rssMB‚ 2690.7734375 disk.nvme0n1.in‚ 1486.0703125! disk.nvme0n1.out‚ 4316.9609375 disk.sda.out‚ 28.46484375 disk.sda1.out‚0( proc.memory.availableMB‚ 248.47265625 disk.drbd2165.out‚0 network.recv‚394614# disk.nvme0c0n1.out‚ 4316.9609375 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0) proc.memory.percent‚16.42317771911621 cpu‚25.852602122331234 network.sent‚ 2.300929e+06 disk.sda3.out‚ 28.46484375$ disk./.usageGB‚589.9036064147949ÆõžG2â- _wandbruntime‚1335  _runtime‚1335í8õºÇãÁ _runtime‚1335.967694173 _step‚123 & lr-AdamW/pg1‚0.0005771014131959182 & lr-AdamW/pg2‚0.0005771014131959182 trainer/global_step‚1549 % _timestamp‚1.7843640574615443e+09ÑBÄoËäÅ & lr-AdamW/pg1‚0.0005771014131959182 & lr-AdamW/pg2‚0.0005771014131959182 trainer/global_step‚1549 % _timestamp‚1.7843640574615443e+09 _runtime‚1335.967694173 _step‚123{=4ô]2å- _wandbruntime‚1335  _runtime‚1335Ø“!Ù'æÓO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6033839 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232852814860288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623201460322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622860910432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1625692491776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2718191 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234048119824384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234048136863744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232852797820928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234048119824384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235673829355520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1625351152640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269Î¥.3Ý'ç×O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14305745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚1335.971827138 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚51986272282624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14295768 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1644670141952 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1625692491776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234048136863744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3315383 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232852797820928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235673470977024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234048119824384 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53630961036800 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235673829355520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235673470977024 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12195458 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1625351152640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6033839 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14305745 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232852814860288 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1623201460322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚124 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235673829355520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1625351152640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1622860910432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1625692491776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2718191 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚234476016320610 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚234048119824384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1644585759232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645593150464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23419ýÄ,àè5377893376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53665294191616 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234623538046316 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1623871964154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1626705125888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2719933 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232999308492800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12203090 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936 _runtime‚1336.597919447 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12203090 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232999325532160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12203092 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚1549 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14304820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3317690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3317408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624212514156 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235821724633600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843640580915363e+09 _step‚125 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52019701041152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362bç‘lé&êãM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235821724633600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843640580915363e+09 _step‚125 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚52019701041152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14314811 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52019719653376 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1645694303232 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1645593150464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234195377893376 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234623180456954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235822083019264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234195377893376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚53665294191616 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6037341 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚234623538046316 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1623871964154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1626705125888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2719933 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232999308492800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12203090 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2719936 _runtime‚1336.597919447 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12203090 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232999325532160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12203092 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚1549 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14304820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14304820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3317690 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3317408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1624212514156}»«_2ë- _wandbruntime‚1336  _runtime‚1336ÖÜeìþ 1loss/FBt1/assignment_loss‚0.08724470528607382 .loss/FRt2/detection_loss‚1.588832476733829 0loss/SRqqt1/detection_loss‚1.312991329318311 1loss/SRqqt2/detection_loss‚0.7345642228716379 _runtime‚ 1336.59916046 trainer/global_step‚1549 % _timestamp‚1.7843640580934741e+09 1loss/FBt2/assignment_loss‚0.05383326980418959 0loss/FRt2/assignment_loss‚1.7303370280848922 /loss/FRt1/detection_loss‚0.6180619008023117 /loss/FBt1/detection_loss‚0.5798498307335346 /loss/FBt2/detection_loss‚1.6486599973656915 &loss/total_loss‚13.096484218486047 epoch‚3 _step‚126 /loss/FRt1/assignment_loss‚3.038491678779203 2loss/SRqqt1/assignment_loss‚0.9862083741585728 2loss/SRqqt2/assignment_loss‚0.7174094045477989ÂnT.í‚ 2loss/SRqqt1/assignment_loss‚0.9862083741585728 2loss/SRqqt2/assignment_loss‚0.7174094045477989 1loss/FBt1/assignment_loss‚0.08724470528607382 .loss/FRt2/detection_loss‚1.588832476733829 0loss/SRqqt1/detection_loss‚1.312991329318311 1loss/SRqqt2/detection_loss‚0.7345642228716379 _runtime‚ 1336.59916046 trainer/global_step‚1549 % _timestamp‚1.7843640580934741e+09 1loss/FBt2/assignment_loss‚0.05383326980418959 0loss/FRt2/assignment_loss‚1.7303370280848922 /loss/FRt1/detection_loss‚0.6180619008023117 /loss/FBt1/detection_loss‚0.5798498307335346 /loss/FBt2/detection_loss‚1.6486599973656915 &loss/total_loss‚13.096484218486047 epoch‚3 _step‚126 /loss/FRt1/assignment_loss‚3.038491678779203~N^8o“î‚(:ˆ §øìÒ´®—í gpu.0.gpu‚83.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ1;ö°ï‚(:¥ §øìÒ‚ŠÀï proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚4323.2685546875) proc.memory.percent‚16.42482280731201 network.recv‚398462 disk.nvme0n1.in‚ 1502.7578125 disk.dm-0.in‚ 1502.7578125$ memory_percent‚98.48508834838867 disk.rbd1.out‚3.546875 disk.sda1.out‚0 disk.sda2.out‚0( disk./.usagePercent‚39.5889423730727 disk.dm-1.out‚28.6875 network.sent‚ 2.304095e+06 disk.sda1.in‚0& proc.memory.availableMB‚ 248.203125 cpu‚25.85082227098865" disk.nvme0c0n1.in‚ 1502.7578125& disk.nvme0c0n1.out‚4323.2685546875 disk.sda.in‚0# proc.memory.rssMB‚ 2691.04296875 disk.sda3.out‚28.6875 disk.dm-1.in‚0 disk.sda2.in‚0$ disk./.usageGB‚589.9038505554199 disk.sda.out‚28.6875 disk.drbd2165.in‚0 disk.drbd2165.out‚0! disk.dm-0.out‚4323.2685546875ú ‚“ð‚(:ˆ ¶øìÒÞàæí gpu.0.gpu‚96.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚196.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.2748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ع/Ôºñ‚(:¯ ¶øìÒ‹ê†ï disk.dm-1.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 248.265625 disk.dm-0.in‚ 1551.015625 network.recv‚402705! disk.nvme0c0n1.in‚ 1551.015625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2690.7734375 disk.sda.in‚0 disk.nvme0n1.in‚ 1551.015625$ disk.nvme0n1.out‚4325.5439453125$ memory_percent‚98.48470687866211 network.sent‚ 2.366945e+06& disk.nvme0c0n1.out‚4325.5439453125 disk.dm-1.out‚ 29.08203125 disk.drbd2165.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚589.9040679931641 disk.drbd2165.in‚0! disk.dm-0.out‚4325.5439453125) proc.memory.percent‚16.42317771911621 cpu‚25.849211824953993 disk.sda2.out‚0 disk.rbd1.out‚3.609375 disk.sda3.out‚ 29.08203125) disk./.usagePercent‚39.58895696550215 disk.sda.out‚ 29.08203125¤ÿt2ò- _wandbruntime‚1377  _runtime‚13773/×WÇóÁ _runtime‚1377.121059996 _step‚127 & lr-AdamW/pg1‚0.0005766651300887848 & lr-AdamW/pg2‚0.0005766651300887848 trainer/global_step‚1599 % _timestamp‚1.7843640986150906e+09Á£ ËôÅ & lr-AdamW/pg1‚0.0005766651300887848 & lr-AdamW/pg2‚0.0005766651300887848 trainer/global_step‚1599 % _timestamp‚1.7843640986150906e+09 _runtime‚1377.121059996 _step‚127_>—2õ- _wandbruntime‚1377  _runtime‚1377mù[Ý 'öÓO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241410171305984 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53657728821248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53657710209024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚243086576338944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1676046654464 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12577058 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3419232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55353434449920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚ 1377.12237508 _step‚128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6224673 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚1599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1673477397340 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55353537444864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14760197 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12577058 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241410171305984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2805441 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14760197 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1676387993600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241851808811868 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243086217960448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14750220 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241852166401118 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240178331414528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚243086217960448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14750220 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241852166401118 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240178331414528 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243086217960448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6224673 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚1599 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1673477397340 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55353537444864 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3418969 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚241410188345344 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14760197 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12577058 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241410171305984 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2805441 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14760197 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1676387993600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438€ÝÊRé2ø- _wandbruntime‚1377  _runtime‚1377 Xõ)ã&ùÝM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12584690 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚1599 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14769263 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12584690 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241999330537462 _step‚129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243234830002688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27343962 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53691157579776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243234471617024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14769263 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240324859125760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241557429374976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1677400627712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1696753062400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12584692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14759272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241557429374976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12584692 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1677400627712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1696753062400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12584692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14759272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241557429374976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12584692 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1674829001064 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚1599 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6228248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14769263 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12584690 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1677059281408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241999330537462 _step‚129 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243234830002688 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27343962 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53691157579776 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27343962 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243234471617024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14769263 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1696837437952 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240324859125760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241557429374976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2©½Ë£2û- _wandbruntime‚1377  _runtime‚1377) Íü /loss/FRt2/detection_loss‚1.4234839359316087 1loss/SRqqt2/detection_loss‚0.7501189054972003 2loss/SRqqt1/assignment_loss‚0.8534847183701478 /loss/FRt2/assignment_loss‚1.740436914194198 1loss/SRqqt1/detection_loss‚1.2296294005463522 /loss/FBt2/detection_loss‚1.2951069406193236 &loss/total_loss‚12.173761367387092 epoch‚3 trainer/global_step‚1599 % _timestamp‚1.7843640992485907e+09 0loss/FRt1/assignment_loss‚2.9742573261390843 0loss/FBt1/detection_loss‚0.46417867012718744 _runtime‚1377.754272975 _step‚130 0loss/SRqqt2/assignment_loss‚0.67992257694901 1loss/FBt1/assignment_loss‚0.12714284177078877 2loss/FBt2/assignment_loss‚0.006080939910317054 /loss/FRt1/detection_loss‚0.6299181973318733 Á\ ý† 0loss/SRqqt2/assignment_loss‚0.67992257694901 1loss/FBt1/assignment_loss‚0.12714284177078877 2loss/FBt2/assignment_loss‚0.006080939910317054 /loss/FRt1/detection_loss‚0.6299181973318733 /loss/FRt2/detection_loss‚1.4234839359316087 1loss/SRqqt2/detection_loss‚0.7501189054972003 2loss/SRqqt1/assignment_loss‚0.8534847183701478 /loss/FRt2/assignment_loss‚1.740436914194198 1loss/SRqqt1/detection_loss‚1.2296294005463522 /loss/FBt2/detection_loss‚1.2951069406193236 &loss/total_loss‚12.173761367387092 epoch‚3 trainer/global_step‚1599 % _timestamp‚1.7843640992485907e+09 0loss/FRt1/assignment_loss‚2.9742573261390843 0loss/FBt1/detection_loss‚0.46417867012718744 _runtime‚1377.754272975 _step‚130‚¼¹rI’þ‚(:‡ ÅøìÒ®²Ûí gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚288.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pa/¯8ÿ‚(:­ ÅøìÒÝî•ï disk.dm-0.in‚ 1551.015625 disk.rbd1.out‚3.609375 cpu‚25.852240983506107 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 29.609375$ disk.nvme0n1.out‚4332.0146484375# proc.memory.rssMB‚ 2738.12109375 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sd‹xÅï€a.out‚ 29.609375 disk.nvme0n1.in‚ 1551.015625* proc.memory.percent‚16.712164878845215 network.recv‚406116& disk.nvme0c0n1.out‚4332.0146484375 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 200.98828125 network.sent‚ 2.371907e+06) disk./.usagePercent‚39.58896694979598 disk.dm-1.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚589.9042167663574! disk.dm-0.out‚4332.0146484375 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 1551.015625$ memory_percent‚98.77326488494873 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 29.609375SŽ]3“€ ‚(:ˆ ÔøìÒЬµî gpu.0.gpu‚97.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚170.776' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%è~¼ ‚(:± ÔøìÒÙˆ¨ó disk.dm-1.in‚0 disk.dm-1.out‚ 30.25390625% disk.nvme0n1.out‚4334.75732421875# memory_percent‚98.7999677658081 disk.nvme0n1.in‚ 1551.015625( proc.memory.availableMB‚ 196.61328125 cpu‚25.894848601441723 disk.rbd1.out‚3.75 disk.drbd2165.out‚0 disk.dm-0.in‚ 1551.015625 disk.sda.in‚0 network.recv‚410558! disk.nvme0c0n1.in‚ 1551.015625# proc.memory.rssMB‚ 2742.41015625 proc.cpu.threads‚17 disk.sda3.out‚ 30.25390625 disk.rbd0.out‚0* proc.memory.percent‚16.738343238830566 disk.sda1.out‚0 network.sent‚ 2.436803e+06 disk.sda2.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚4334.75732421875 disk.sda.out‚ 30.25390625$ disk./.usageGB‚589.9041328430176 disk.sda2.in‚0' disk.nvme0c0n1.out‚4334.75732421875 disk.sda3.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚39.58896131763023 disk.sda1.in‚0<”¤2‚ - _wandbruntime‚1395  _runtime‚13956£ÙM=ƒ Çz /particle/accuracy_4_of_4‚0.9493328928947449 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06858547776937485 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6453205943107605 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6075911521911621 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6045793890953064 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7168178558349609 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05282276123762131 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6351840496063232 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5868291854858398 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1770833432674408 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5728319883346558 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6048753261566162 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 % _timestamp‚1.7843641166921704e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7188208699226379 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13213253021240234 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6351840496063232 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06354586035013199 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.39405059814453125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6993038654327393 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3701649308204651 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7331820726394653 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954084753990173 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6292433142662048 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_3_of_5‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35726675391197205 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16434915363788605 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9881314039230347 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.735345184803009 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5177947282791138 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1895800083875656 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 jet/accuracy_5_of_5‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9900687336921692 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.061806611716747284 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6358088254928589 /particle/accuracy_3_of_3‚0.9995090961456299 _step‚131 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8234471082687378 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6814137697219849 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36507779359817505 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.74692702293396 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7718409299850464 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4513888955116272 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06558717787265778 *loss/val_total_loss‚11.695467155517088 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.513702392578125 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.049516625702381134 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06558717787265778 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9952826499938965 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9884859919548035 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07516428083181381 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.002043274464085698 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7336156964302063 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.41123032569885254 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7768990397453308 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7722681164741516 ,jet/accuracy_4_of_4‚0.007130124606192112 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6253743171691895 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.721380352973938 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012228101259097457 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05963461846113205 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.22027342021465302 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8760308623313904 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976553320884705 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07067372649908066 *jet/accuracy_1_of_1‚0.2615967094898224 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011023205704987049 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7138321995735168 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909943342208862 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08855537325143814 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7908625602722168 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966614842414856 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6842547059059143 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.07977890968322754 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6763771772384644 (particle/accuracy‚0.8330814838409424 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 *jet/accuracy_1_of_3‚0.9058194160461426 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0949999988079071 +particle/sensitivity‚0.5997878909111023 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.11963566392660141 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982918500900269 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4833333492279053 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896191954612732 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7282124161720276 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.055124133825302124 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_4‚0.9581987261772156 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6505467891693115 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9769719243049622 'particle/f_score‚0.6068070530891418 ( Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.04475920647382736 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08201577514410019 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6594220399856567 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6955973505973816 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9191066026687622 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11241084337234497 epoch‚3 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0554397776722908 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.35976642370224 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06954680383205414 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7762839198112488 *jet/accuracy_1_of_2‚0.7681130766868591 +jet/accuracy_2_of_3‚0.43932032585144043 jet/accuracy_3_of_5‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.0904393345117569 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0696721151471138 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06879837065935135 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6320782899856567 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4626275897026062 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09773605316877365 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960986375808716 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7448534369468689 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09535295516252518 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 trainer/global_step‚1599 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941112995147705 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6156967878341675 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.8968788981437683 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8187806010246277 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874308705329895 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.055124133825302124 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3064229488372803 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01575545407831669 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34259259700775146 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947565793991089 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_3_of_4‚0.9970238208770752 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16132065653800964 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06873919069766998 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08088580518960953 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5973641276359558 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005122151225805283 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.0766725167632103 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.29199427366256714 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 _runtime‚1395.199136439 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7validation_average_jet_accuracy‚0.31276649236679077 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.793903648853302 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.10473285615444183 +jet/accuracy_3_of_3‚0.04030393064022064 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_3‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22503754496574402 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.22767594456672668 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009620046243071556 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06274937093257904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909065961837769 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6205493807792664 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4638465642929077 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34375 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6993833780288696 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6797575950622559 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7908625602722168 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6244134306907654 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06906410306692123 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6653257608413696 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04242125153541565 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1z¯ñˆ:„ Ìz 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0554397776722908 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.35976642370224 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06954680383205414 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7762839198112488 *jet/accuracy_1_of_2‚0.7681130766868591 +jet/accuracy_2_of_3‚0.43932032585144043 jet/accuracy_3_of_5‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.0904393345117569 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.0696721151471138 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06879837065935135 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6320782899856567 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4626275897026062 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09773605316877365 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960986375808716 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7448534369468689 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6071428656578064 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09535295516252518 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 trainer/global_step‚1599 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9941112995147705 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6156967878341675 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.8968788981437683 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8187806010246277 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874308705329895 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.055124133825302124 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3064229488372803 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01575545407831669 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34259259700775146 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9947565793991089 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_3_of_4‚0.9970238208770752 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16132065653800964 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06873919069766998 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08088580518960953 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5973641276359558 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.005122151225805283 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.0766725167632103 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.29199427366256714 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 _runtime‚1395.199136439 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7validation_average_jet_accuracy‚0.31276649236679077 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.793903648853302 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.10473285615444183 +jet/accuracy_3_of_3‚0.04030393064022064 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_1_of_3‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.22503754496574402 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.22767594456672668 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.009620046243071556 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.06274937093257904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909065961837769 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6205493807792664 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4638465642929077 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34375 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6993833780288696 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6797575950622559 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7908625602722168 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6244134306907654 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06906410306692123 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6653257608413696 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04242125153541565 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9493328928947449 particle/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06858547776937485 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6453205943107605 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6075911521911621 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6045793890953064 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7168178558349609 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.05282276123762131 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6351840496063232 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5868291854858398 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1770833432674408 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5728319883346558 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6048753261566162 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 % _timestamp‚1.7843641166921704e+09 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7188208699226379 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13213253021240234 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6351840496063232 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06354586035013199 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.39405059814453125 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6993038654327393 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3701649308204651 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7331820726394653 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9954084753990173 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6292433142662048 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_3_of_5‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35726675391197205 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16434915363788605 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9881314039230347 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.735345184803009 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5177947282791138 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1895800083875656 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 jet/accuracy_5_of_5‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9900687336921692 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.061806611716747284 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6358088254928589 /particle/accuracy_3_of_3‚0.9995090961456299 _step‚131 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8234471082687378 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6814137697219849 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36507779359817505 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.74692702293396 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7718409299850464 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4513888955116272 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06558717787265778 *loss/val_total_loss‚11.695467155517088 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.513702392578125 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.049516625702381134 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06558717787265778 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9952826499938965 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9884859919548035 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.07516428083181381 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.002043274464085698 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7336156964302063 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.41123032569885254 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7768990397453308 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7722681164741516 ,jet/accuracy_4_of_4‚0.007130124606192112 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6253743171691895 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.721380352973938 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012228101259097457 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.05963461846113205 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.22027342021465302 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8760308623313904 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976553320884705 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.07067372649908066 *jet/accuracy_1_of_1‚0.2615967094898224 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011023205704987049 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7138321995735168 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9909943342208862 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08855537325143814 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7908625602722168 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966614842414856 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6842547059059143 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.07977890968322754 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6763771772384644 (particle/accuracy‚0.8330814838409424 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 *jet/accuracy_1_of_3‚0.9058194160461426 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.0949999988079071 +particle/sensitivity‚0.5997878909111023 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.11963566392660141 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9982918500900269 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4833333492279053 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896191954612732 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7282124161720276 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.055124133825302124 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_4‚0.9581987261772156 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6505467891693115 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9769719243049622 'particle/f_score‚0.6068070530891418 ( Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy\À¶æÊ_3_of_4‚0.04475920647382736 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08201577514410019 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6594220399856567 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6955973505973816 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9191066026687622 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.11241084337234497 epoch‚3ƒbè ‘… ‚(:† ãøìÒžîßí gpu.0.gpu‚22.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚162.065' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.413 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼îï°­† ‚(:¢ ãøìÒŠ‹¶ï disk.sda3.in‚0$ memory_percent‚95.46520709991455 network.recv‚413988 disk.rbd0.out‚0 disk.sda.in‚0) proc.memory.percent‚16.42482280731201$ disk./.usageGB‚589.9052696228027 disk.rbd1.out‚3.75( proc.memory.availableMB‚ 742.98046875) disk./.usagePercent‚39.58903760787543 disk.rbd1.in‚0 disk.sda.out‚30.625# proc.memory.rssMB‚ 2691.04296875 network.sent‚ 2.439725e+06 disk.drbd2165.in‚0% disk.nvme0n1.out‚4585.86767578125 disk.rbd0.in‚0 disk.dm-0.in‚ 1551.015625" disk.dm-0.out‚4585.86767578125 disk.sda1.in‚0 disk.sda3.out‚30.625 cpu‚25.891389211213856' disk.nvme0c0n1.out‚4585.86767578125 disk.dm-1.out‚30.625 disk.nvme0n1.in‚ 1551.015625 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 1551.015625 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.out‚0Þ¼Ö‡ ‚(:’ òøìÒºšî gpu.0.gpu‚56.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼‡ÙYÀˆ ‚(:µ òøìÒÀɘï disk.sda.in‚0 disk.sda3.out‚ 30.88671875 disk.dm-1.out‚ 30.88671875 disk.drbd2165.in‚0) disk./.usagePercent‚39.58898333427817 network.recv‚418365 disk.rbd0.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 1653.23828125' disk.nvme0c0n1.out‚4834.52685546875 proc.cpu.threads‚17 disk.dm-0.in‚ 1653.23828125" disk.dm-0.out‚4834.52685546875 disk.sda3.in‚0' proc.memory.availableMB‚ 742.7734375# proc.memory.rssMB‚ 2691.04296875 disk.rbd0.in‚0! disk.nvme0n1.in‚ 1653.23828125$ disk./.usageGB‚589.9044609069824% disk.nvme0n1.out‚4834.52685546875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.out‚ 30.88671875 disk.sda1.in‚0) proc.memory.percent‚16.42482280731201 cpu‚25.889245205157227$ memory_percent‚95.46647071838379 network.sent‚ 2.49702e+06 disk.rbd1.out‚3.75™ùe02‰ - _wandbruntime‚1437  _runtime‚1437~¡0©ÆŠ À trainer/global_step‚1649 $ _timestamp‚1.784364159178363e+09 _runtime‚1437.685956087 _step‚132 & lr-AdamW/pg1‚0.0005762104999735266 & lr-AdamW/pg2‚0.0005762104999735266p¾OìË‹ Å & lr-AdamW/pg1‚0.0005762104999735266 & lr-AdamW/pg2‚0.0005762104999735266 trainer/global_step‚1649 $ _timestamp‚1.784364159178363e+09 _runtime‚1437.685956087 _step‚132„`Îï2Œ - _wandbruntime‚1437  _runtime‚1437Úü?µÚ' ÔO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744475364352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553589 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57110232629760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079326 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55345060235264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13079326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252446623180800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚1437.686069752 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¨ƒÜ7ß'Ž ÙO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13079324 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250701806477312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249412545261568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚250701789437952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250701789437952 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚1649 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2895211 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13079326 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252446264802304 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252446623180800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚1437.686069752 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1744475364352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3553589 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57110232629760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13079326 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚55345060235264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57144572479488 _runtime‚1438.318800132 _step‚134 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55378488993792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_b/ƒatch_end/active_bytes.all.allocated‚252594876844544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2896956 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250849030467584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13086958 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚250849047506944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360·¨¥‘è&‘ âM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250849030467584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252594876844544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249559038894080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13086958 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6452405 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2896953 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13086956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13086956 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6452663 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚57144572479488 _runtime‚1438.318800132 _step‚134 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚55378488993792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252594876844544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2896956 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15294042 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250849030467584 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13086958 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚250849047506944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15284051 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3555452 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ER„¯2’ - _wandbruntime‚1438  _runtime‚1438å œ“  0loss/FBt1/detection_loss‚0.43137363645497057 /loss/FRt1/assignment_loss‚2.922622522521526 0loss/FRt2/assignment_loss‚1.6912042060077073 2loss/SRqqt2/assignment_loss‚0.6310947646627356 0loss/SRqqt1/detection_loss‚1.226404223415368 1loss/SRqqt2/detection_loss‚0.7739841054516057 trainer/global_step‚1649 % _timestamp‚1.7843641598142571e+09 &loss/total_loss‚11.933341865650583 _runtime‚1438.319941418 /loss/FBt2/detection_loss‚1.0615609275798004 epoch‚4 1loss/FBt1/assignment_loss‚0.10853105058495781 1loss/FBt2/assignment_loss‚0.12856306567977624 /loss/FRt2/detection_loss‚1.5093307300402803 _step‚135 2loss/SRqqt1/assignment_loss‚0.8636605864388535 /loss/FRt1/detection_loss‚0.5850120468130007KæÛ ” † /loss/FRt1/assignment_loss‚2.922622522521526 0loss/FRt2/assignment_loss‚1.6912042060077073 2loss/SRqqt2/assignment_loss‚0.6310947646627356 0loss/SRqqt1/detection_loss‚1.226404223415368 1loss/SRqqt2/detection_loss‚0.7739841054516057 trainer/global_step‚1649 % _timestamp‚1.7843641598142571e+09 &loss/total_loss‚11.933341865650583 _runtime‚1438.319941418 /loss/FBt2/detection_loss‚1.0615609275798004 epoch‚4 1loss/FBt1/assignment_loss‚0.10853105058495781 1loss/FBt2/assignment_loss‚0.12856306567977624 /loss/FRt2/detection_loss‚1.5093307300402803 _step‚135 2loss/SRqqt1/assignment_loss‚0.8636605864388535 /loss/FRt1/detection_loss‚0.5850120468130007 0loss/FBt1/detection_loss‚0.43137363645497057‡Ælùr‘• ‚(:† ùìÒÊÁ©í gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì…—Å– ‚(:º ùìÒø¦ñï disk.sda.in‚0 disk.sda3.out‚ 31.10546875 disk.dm-1.in‚0( proc.memory.availableMB‚ 742.55078125$ disk./.usageGB‚589.9042472839355 disk.sda2.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 1653.23828125 cpu‚25.887111107361417 disk.rbd1.out‚3.8125) proc.memory.percent‚16.42482280731201 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 1653.23828125' disk.nvme0c0n1.out‚4839.66845703125 disk.rbd1.in‚0 disk.sda.out‚ 31.10546875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2691.04296875 network.recv‚421723 disk.dm-0.in‚ 1653.23828125" disk.dm-0.out‚4839.66845703125 disk.dm-1.out‚ 31.10546875 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 2.501932e+06* disk./.usagePercent‚39.588968997856256% disk.nvme0n1.out‚4839.66845703125$ memory_percent‚95.46782970428467ñÇ—éž— ‚(:“ ùìÒ¨ŒÍï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æûâ‡Á˜ ‚(:¶ ùìÒÅÕõð# proc.memory.rssMB‚ 2691.04296875# disk.nvme0c0n1.in‚ 1653.23828125 disk.sda.in‚0 disk.dm-0.in‚ 1653.23828125 disk.rbd1.out‚3.8125) proc.memory.percent‚16.42482280731201 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 2.567074e+06$ disk.nvme0n1.out‚4842.2958984375 network.recv‚426095 disk.sda3.in‚0 disk.dm-1.out‚ 31.34765625 disk.rbd0.out‚0! disk.nvme0n1.in‚ 1653.23828125 proc.cpu.threads‚17& disk.nvme0c0n1.out‚4842.2958984375 disk.sda2.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚589.9044494628906 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚4842.2958984375 disk.sda2.in‚0 cpu‚25.88534565383355 disk.sda1.in‚0* disk./.usagePercent‚39.588982566255574 disk.sda.out‚ 31.34765625 disk.sda3.out‚ 31.34765625$ memory_percent‚95.46868801116943( proc.memory.availableMB‚ 742.41015625zÒý“™ ‚(:ˆ ŸùìÒÀÊî gpu.0.gpu‚58.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04sçš ‚(:· ŸùìÒ‰‡Øï disk.sda1.in‚0 network.sent‚ 2.570097e+06 disk.sda1.out‚0 disk.dm-1.out‚ 31.6328125 disk.sda2.in‚0$ memory_percent‚95.46725749969482) disk./.usagePercent‚39.58897463002201 disk.sda3.in‚0 disk.sda3.out‚ 31.6328125) proc.memory.percent‚16.42482280731201" disk.dm-0.out‚4847.23681640625 disk.sda.out‚ 31.6328125 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 1653.23828125 disk.dm-0.in‚ 1653.23828125 cpu‚25.88353045328438 proc.cpu.threads‚17$ disk./.usageGB‚589.9043312072754' disk.nvme0c0n1.out‚4847.23681640625 disk.drbd2165.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 742.64453125# proc.memory.rssMB‚ 2691.04296875 disk.rbd0.out‚0 disk.sda2.out‚0 network.recv‚429691 disk.rbd1.out‚3.890625 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚4847.23681640625# disk.nvme0c0n1.in‚ 1653.23828125ŒÑ®÷2› - _wandbruntime‚1478  _runtime‚1478Ò›MZÆœ À & lr-AdamW/pg2‚0.0005757375520501866 trainer/global_step‚1699 $ _timestamp‚1.784364200304833e+09 _runtime‚1478.810820009 _step‚136 & lr-AdamW/pg1‚0.0005757375520501866Æ9ÌUË Å & lr-AdamW/pg2‚0.0005757375520501866 trainer/global_step‚1699 $ _timestamp‚1.784364200304833e+09 _runtime‚1478.810820009 _step‚136 & lr-AdamW/pg1‚0.0005757375520501866ˆH(2ž - _wandbruntime‚1478  _runtime‚1478c\¬Ù'Ÿ ÓO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1792430096390 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13460926 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460924 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58832910766592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1816393992704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15739428 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2982461 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3658076 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1795170866176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256738078855168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1816305415680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256738061815808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0O9²  ØO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚1478.815818606 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259859370164224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259859011785728 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _step‚137 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57016516773888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1792430096390 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13460926 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15729451 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13460924 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚58832910766592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1816393992704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15739428 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2982461 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3658076 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1795170866176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256738078855168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15739428 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15729451 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1816305415680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259859011785728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256738061815808 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1795170866176 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3658328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚1699 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13468556 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258211081949184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1796524839424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1796183493120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58867365412864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57049926920192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258211081949184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1796183493120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3660136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817419880448 % _timestamp‚1.7843642009396248e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1796524839424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1479.445876113 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258678371227154 _step‚138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258211098988544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1793781700114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29207059 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468556 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15738503 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3660392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256884589527040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15748494 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304*q‘1é&£ ãM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1793781700114 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29207059 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256884572487680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13468556 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15738503 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3660392 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256884589527040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15748494 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58867264522240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚57049945532416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1793441150112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29217052 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚1699 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13468556 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258211081949184 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1796524839424 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1796183493120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58867365412864 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57049926920192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258211081949184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1796183493120 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6644339 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3660136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817419880448 % _timestamp‚1.7843642009396248e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1796524839424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1479.445876113 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258678371227154 _step‚138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258211098988544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360Šò,2¤ - _wandbruntime‚1479  _runtime‚1479ÿCÏ¥ û .loss/FBt2/detection_loss‚1.780374591740278 $ _timestamp‚1.784364200941371e+09 1loss/SRqqt2/detection_loss‚0.7313496980312583 .loss/FBt1/detection_loss‚0.640062165859057 _step‚139 2loss/SRqqt1/assignment_loss‚0.8256033520715741 1loss/FBt1/assignment_loss‚0.08204290540121133 -loss/FRt1/detection_loss‚0.63984857326634 /loss/FRt2/detection_loss‚1.4916735268488992 1loss/SRqqt1/detection_loss‚1.4412883792645657 trainer/global_step‚1699 _runtime‚1479.447064407 /loss/FRt1/assignment_loss‚2.901136704825188 0loss/FRt2/assignment_loss‚1.6024099685909117 1loss/FBt2/assignment_loss‚0.13887645929711512 %loss/total_loss‚12.99910817071814 epoch‚4 2loss/SRqqt2/assignment_loss‚0.7244418455217404¸ä*¦ € 2loss/SRqqt1/assignment_loss‚0.8256033520715741 1loss/FBt1/assignment_loss‚0.08204290540121133 -loss/FRt1/detection_loss‚0.63984857326634 /loss/FRt2/detection_loss‚1.4916735268488992 1loss/SRqqt1/detection_loss‚1.4412883792645657 trainer/global_step‚1699 _runtime‚1479.447064407 /loss/FRt1/assignment_loss‚2.901136704825188 0loss/FRt2/assignment_loss‚1.6024099685909117 1loss/FBt2/assignment_loss‚0.13887645929711512 %loss/total_loss‚12.99910817071814 epoch‚4 2loss/SRqqt2/assignment_loss‚0.7244418455217404 .loss/FBt2/detection_loss‚1.780374591740278 $ _timestamp‚1.784364200941371e+09 1loss/SRqqt2/detection_loss‚0.7313496980312583 .loss/FBt1/detection_loss‚0.640062165859057 _step‚139‹®Ê×鞧 ‚(:“ ®ùìÒꀅï gpu.0.gpu‚86.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.197' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.439400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"­q÷¤¨ ‚(:™ ®ùìÒ†ËÈð disk.rbd0.in‚0 disk.nvme0n1.out‚ 4850.90625 disk.sda1.out‚0! disk.nvme0n1.in‚ 1754.05078125 disk.dm-0.in‚ 1754.05078125! disk.nvme0c0n1.out‚ 4850.90625$ memory_percent‚95.46713829040527 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2691.04296875) proc.memory.percent‚16.42482280731201 proc.cpu.threads‚17 network.recv‚432736 disk.sda3.out‚32.125 disk.dm-1.in‚0 cpu‚25.881343756760565) disk./.usagePercent‚39.58896387770557 disk.sda3.in‚0 disk.dm-1.out‚32.125$ disk./.usageGB‚589.9041709899902 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1754.05078125 disk.dm-0.out‚ 4850.90625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚32.125 network.sent‚ 2.575086e+06 disk.rbd0.out‚0 disk.rbd1.out‚3.890625' proc.memory.availableMB‚ 742.6640625ö_møž© ‚(:“ ½ùìÒâø–î gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚253.165' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.632999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dŸÁ±Ãª ‚(:¸ ½ùìÒ”–Èï disk.rbd1.in‚0 disk.sda3.in‚0) proc.memory.percent‚16.42482280731201$ disk./.usageGB‚589.9042053222656 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 network.sent‚ 2.637874e+06 network.recv‚437412" disk.dm-0.out‚4855.70458984375 disk.sda2.out‚0# proc.memory.rssMB‚ 2691.04296875 proc.cpu.threads‚17) disk./.usagePercent‚39.58896618177338 disk.drbd2165.out‚0 disk.dm-0.in‚ 1754.05078125 disk.rbd1.out‚3.953125 disk.sda1.out‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚4855.70458984375& proc.memory.availableMB‚ 742.734375 cpu‚25.87959552006886 disk.drbd2165.in‚0 disk.sda3.out‚ 32.48046875 disk.dm-1.out‚ 32.48046875! disk.nvme0n1.in‚ 1754.05078125% disk.nvme0n1.out‚4855.70458984375 disk.sda2.in‚0 disk.sda.in‚0# disk.nvme0c0n1.in‚ 1754.05078125 disk.sda.out‚ 32.48046875$ memory_percent‚95.46670913696289ã•ž« ‚(:“ ÌùìÒè–®í gpu.0.gpu‚45.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚131.064' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.212799999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óC0Ĭ ‚(:¹ ÌùìÒœÄï disk.rbd0.out‚0( proc.memory.availableMB‚ 742.73046875 network.recv‚440512 disk.sda2.out‚0 disk.rbd1.in‚0# disk./.usageGB‚589.904167175293 disk.sda3.out‚ 32.68359375 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚4858.76123046875! disk.nvme0n1.in‚ 1754.05078125# disk.nvme0c0n1.in‚ 1754.05078125 network.sent‚ 2.642678e+06 disk.sda2.in‚0 disk.dm-1.out‚ 32.68359375# memory_percent‚95.4667329788208 proc.cpu.threads‚17 disk.sda3.in‚0% disk.nvme0n1.out‚4858.76123046875 disk.dm-0.in‚ 1754.05078125" disk.dm-0.out‚4858.76123046875 disk.sda.out‚ 32.68359375 disk.sda1.in‚0 disk.rbd1.out‚3.953125 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2691.04296875) proc.memory.percent‚16.42482280731201 cpu‚25.87784709633585* disk./.usagePercent‚39.588963621698035ÅúÀ´2­ - _wandbruntime‚1519  _runtime‚1519åzýªÅ® ¿ % lr-AdamW/pg1‚0.000575246316695326 % lr-AdamW/pg2‚0.000575246316695326 trainer/global_step‚1749 % _timestamp‚1.7843642412109816e+09 _runtime‚1519.716968392 _step‚140¼˜º‡Ê¯ Ä % lr-AdamW/pg1‚0.000575246316695326 % lr-AdamW/pg2‚0.000575246316695326 trainer/global_step‚1749 % _timestamp‚1.7843642412109816e+09 _runtime‚1519.716968392 _step‚140Œ:Ò3R2° - _wandbruntime‚1519  _runtime‚1519€ýÄÔu± ÔO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1867563070976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3069711 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1749 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60555604251136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16183903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1845866368000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚141 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1845866368000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265425909440512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6831259 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265425892401152 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6831259 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267271758769152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265425892401152 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1843046583298 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1846207707136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16183903 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1867563070976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3069711 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1749 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚60555604251136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16183903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101>!ùÉ2³ - _wandbruntime‚1520  _runtime‚1520ºÓxä&´ ÞM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60590059317248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266054163718300 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60589958426624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58721364846592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1868675858432 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267420012425728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16192955 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1844057637020 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16202946 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13850158 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264210123120640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1846878994944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265573133430784 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267420370811392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1846878994944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265573133430784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002î¬D¦ƒ µ ãM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚267420370811392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1846878994944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265573133430784 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 trainer/global_step‚1749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6835116 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60590059317248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚266054163718300 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13850158 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60589958426624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58721364846592 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1868675858432 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30053104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚267420012425728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16192955 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1844057637020 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16202946 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚267420370811392 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30043111 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13850158 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264210123120640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1846878994944 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3071453 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3763920 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3763663 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264210106081280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265573150470144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265573133430784 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088Ž¢¸R2¶ - _wandbruntime‚1520  _runtime‚1520FëY· € 1loss/FBt2/assignment_loss‚0.03477904898886718 /loss/FRt1/detection_loss‚0.6524468164164001 trainer/global_step‚1749 _runtime‚1520.352490298 /loss/FBt2/detection_loss‚1.2119790770304508 epoch‚4 %loss/total_loss‚11.89000409426182 /loss/FRt2/assignment_loss‚1.552744021305459 2loss/SRqqt2/assignment_loss‚0.7001464465158737 1loss/FBt1/assignment_loss‚0.10854694658232505 /loss/FRt2/detection_loss‚1.5110415467554579 1loss/SRqqt2/detection_loss‚0.7659350973728203 2loss/SRqqt1/assignment_loss‚0.7571010027303348 1loss/SRqqt1/detection_loss‚1.2308192834389367 % _timestamp‚1.7843642418467014e+09 _step‚143 /loss/FBt1/detection_loss‚0.5937867010073328 0loss/FRt1/assignment_loss‚2.7706781061175634®£ƒË ¸ … 1loss/SRqqt1/detection_loss‚1.2308192834389367 % _timestamp‚1.7843642418467014e+09 _step‚143 /loss/FBt1/detection_loss‚0.5937867010073328 0loss/FRt1/assignment_loss‚2.7706781061175634 1loss/FBt2/assignment_loss‚0.03477904898886718 /loss/FRt1/detection_loss‚0.6524468164164001 trainer/global_step‚1749 _runtime‚1520.352490298 /loss/FBt2/detection_loss‚1.2119790770304508 epoch‚4 %loss/total_loss‚11.89000409426182 /loss/FRt2/assignment_loss‚1.552744021305459 2loss/SRqqt2/assignment_loss‚0.7001464465158737 1loss/FBt1/assignment_loss‚0.10854694658232505 /loss/FRt2/detection_loss‚1.5110415467554579 1loss/SRqqt2/detection_loss‚0.7659350973728203 2loss/SRqqt1/assignment_loss‚0.7571010027303348lt+\ž¹ ‚(:“ ÛùìÒŸµæï gpu.0.gpu‚54.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wý1ªº ‚(:Ÿ ÛùìÒï¡Œò) disk./.usagePercent‚39.58896720580351# disk.nvme0c0n1.in‚ 1754.05078125 disk.dm-1.out‚33 disk.drbd2165.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 742.49609375 disk.sda.in‚0 disk.rbd1.out‚4.03125 disk.dm-1.in‚0 network.recv‚443984 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.out‚33" disk.dm-0.out‚4864.55126953125) proc.memory.percent‚16.42482280731201 disk.sda3.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 1754.05078125 disk.dm-0.in‚ 1754.05078125 cpu‚25.876042482056487 network.sent‚ 2.660765e+06 disk.sda2.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚4864.55126953125$ memory_percent‚95.46816349029541' disk.nvme0c0n1.out‚4864.55126953125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0$ disk./.usageGB‚589.9042205810547 disk.sda.out‚33# proc.memory.rssMB‚ 2691.04296875'ÿW*ž» ‚(:“ êùìҺǙí gpu.0.gpu‚87.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:h4˜¶¼ ‚(:« êùìÒ´ü°ï disk.rbd1.in‚0 disk.sda.out‚ 33.296875 disk.sda1.in‚0 disk.dm-1.out‚ 33.296875 disk.rbd0.out‚0$ disk.nvme0n1.out‚4869.2451171875# memory_percent‚95.4660177230835) proc.memory.percent‚16.42482280731201 disk.drbd2165.out‚0 network.recv‚447428( disk./.usagePercent‚39.5889625976679 disk.drbd2165.in‚0 network.sent‚ 2.709557e+06$ disk./.usageGB‚589.9041519165039 disk.sda2.in‚0 disk.rbd1.out‚4.03125" disk.nvme0c0n1.in‚ 1864.5546875 disk.sda1.out‚0 disk.sda3.out‚ 33.296875# proc.memory.rssMB‚ 2691.04296875& disk.nvme0c0n1.out‚4869.2451171875 disk.rbd0.in‚0 disk.dm-0.in‚ 1864.5546875! disk.dm-0.out‚4869.2451171875 cpu‚25.87429415130376 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 1864.5546875 disk.dm-1.in‚0( proc.memory.availableMB‚ 742.84765625#€ ‘½ ‚(:† ùùìҴܵí gpu.0.gpu‚9.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´¸¾ ‚(:­ ùùìÒò¸±ð disk.sda1.in‚0( proc.memory.availableMB‚ 496.41796875 network.recv‚451004 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1864.5546875# proc.memory.rssMB‚ 2691.04296875$ disk./.usageGB‚589.9040603637695 disk.dm-1.out‚33.59375 disk.rbd1.in‚0$ memory_percent‚96.97010517120361) proc.memory.percent‚16.42482280731201 proc.cpu.threads‚17) disk./.usagePercent‚39.58895645348708 disk.sda3.out‚33.59375 disk.dm-1.in‚0 disk.sda2.in‚0 cpu‚25.87245301673181 disk.sda1.out‚0% disk.nvme0n1.out‚4875.64306640625 disk.drbd2165.out‚0" disk.dm-0.out‚4875.64306640625 disk.rbd1.out‚4.15625 network.sent‚ 2.714477e+06 disk.nvme0n1.in‚ 1864.5546875" disk.nvme0c0n1.in‚ 1864.5546875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚33.59375 disk.sda2.out‚0' disk.nvme0c0n1.out‚4875.64306640625 disk.sda.in‚0;xN2¿ - _wandbruntime‚1560  _runtime‚1560멯ÆÀ À _runtime‚1560.849408272 _step‚144 & lr-AdamW/pg1‚0.0005747368254600737 & lr-AdamW/pg2‚0.0005747368254600737 trainer/global_step‚1799 $ _timestamp‚1.784364282343429e+09Â4íœËÁ Å & lr-AdamW/pg1‚0.0005747368254600737 & lr-AdamW/pg2‚0.0005747368254600737 trainer/global_step‚1799 $ _timestamp‚1.784364282343429e+09 _runtime‚1560.849408272 _step‚144cÖéM2 - _wandbruntime‚1560  _runtime‚1560Œ•I$Ú'à ÔO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16638355 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3864732 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918800508928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1896903208960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14224124 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271389129003008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚1799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60359392626688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3156961 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273283149662464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274684864131072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60359374014464 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16648332 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3864982 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1894003620096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274684505752576 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7021690 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271389146042368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14224126 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚62278279615488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ·6Ç3%Ä ÙO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1896561869824 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16638355 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3864732 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1918800508928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1896903208960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14224124 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271389129003008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚1799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚60359392626688 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16638355 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3156961 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚273283149662464 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274684864131072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274684505752576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60359374014464 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16648332 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16648332 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3864982 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚272787960922112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1894003620096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274684505752576 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7021690 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271389146042368 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1896903208960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚272787960922112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14224126 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14224126 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.all•fײ¬ocated‚62278279615488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591‘HãE2Å - _wandbruntime‚1561  _runtime‚1561¶ÊmEã&Æ ÝM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894674123928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60392802772992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895014673930 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3158703 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16647407 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62312638620160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272935201951744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272935184912384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1897574496768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274832759409152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272935184912384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7025670 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16657398 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710ƒ—Àè&Ç âM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚274833117794816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1897574496768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚273430671388170 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 % _timestamp‚1.7843642829733613e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1897574496768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274832759409152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1919916028416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚272935184912384 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7025670 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16657398 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3866710 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚146 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7025413 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271535656714240 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1894674123928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60392802772992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1895014673930 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3158703 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16647407 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚274833117794816 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚62312638620160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14231758 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272935201951744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272935184912384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16657398 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14605726 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280658942153722 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1944620107004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282097252736000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14605726 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280150012403712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17102784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62030830553088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969948624384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000760565248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7212906 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17102784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280149995364352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17092807 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 _step‚149 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1947257371648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278714679635968 % _timestamp‚1.7843643233755133e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3244208 SEDeviceStatsMonitor.on_„¼Æ train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278714662596608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3244211 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17102784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚62030830553088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1969948624384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64000760565248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7212906 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17102784 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚280149995364352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280149995364352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17092807 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚280659299742972 _step‚149 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1947257371648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3968695 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278714679635968 % _timestamp‚1.7843643233755133e+09 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3244208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278714662596608 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3244211 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282097252736000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14605726 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64000867754496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚280658942153722 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1944620107004 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282097252736000 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282097611114496 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14605726 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚280150012403712 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7216682 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971057270784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280806821468678 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280806463879316 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17111850 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280297253433344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280297236393984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1945290610836 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3245953 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17101859 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1948269998592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚280297236393984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1948611344896 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14613356 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31715215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14613356 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282245864778240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31715215 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3245956 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62064259311616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282245864778240 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7216429 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14613358 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17101859 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62064240699392 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7216682 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1971057270784 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚280806821468678 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚280806463879316 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17111850 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚282245864778240 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280297253433344 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚64035316582400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280297236393984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1945290610836 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3245953 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17101859 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UCDeviceStatsMoZM¤·Çnitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1948269998592 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64035209400320 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169–ž™é2Ø - _wandbruntime‚1602  _runtime‚1602ó7¢Ù ÿ 2loss/SRqqt1/assignment_loss‚0.8738158702076847 1loss/SRqqt1/detection_loss‚1.0708211300592767 &loss/total_loss‚11.731299904470095 % _timestamp‚1.7843643240041802e+09 _step‚151 /loss/FRt1/assignment_loss‚2.569508121039841 2loss/SRqqt2/assignment_loss‚0.6402665381691107 1loss/FBt2/assignment_loss‚0.13003058276070353 /loss/FRt2/detection_loss‚1.4604358031922766 .loss/FBt2/detection_loss‚1.255118513527589 _runtime‚1602.509896052 epoch‚4 0loss/FRt2/assignment_loss‚1.5104920765143315 1loss/FBt1/assignment_loss‚0.12895752749123174 /loss/FRt1/detection_loss‚0.6719678054016485 1loss/SRqqt2/detection_loss‚0.7402638156532654 .loss/FBt1/detection_loss‚0.679622120453135 trainer/global_step‚1849¬F™Ó Ú „ 2loss/SRqqt1/assignment_loss‚0.8738158702076847 1loss/SRqqt1/detection_loss‚1.0708211300592767 &loss/total_loss‚11.731299904470095 % _timestamp‚1.7843643240041802e+09 _step‚151 /loss/FRt1/assignment_loss‚2.569508121039841 2loss/SRqqt2/assignment_loss‚0.6402665381691107 1loss/FBt2/assignment_loss‚0.13003058276070353 /loss/FRt2/detection_loss‚1.4604358031922766 .loss/FBt2/detection_loss‚1.255118513527589 _runtime‚1602.509896052 epoch‚4 0loss/FRt2/assignment_loss‚1.5104920765143315 1loss/FBt1/assignment_loss‚0.12895752749123174 /loss/FRt1/detection_loss‚0.6719678054016485 1loss/SRqqt2/detection_loss‚0.7402638156532654 .loss/FBt1/detection_loss‚0.679622120453135 trainer/global_step‚1849—–‘ “Û ‚(:ˆ ¦úìÒÉî’í gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ€j¥¶Ü ‚(:« ¦úìҜ̡ð disk.sda.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0# proc.memory.availableMB‚496.375 disk.rbd1.out‚4.234375 disk.sda.out‚ 34.64453125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 1940.359375 disk.sda2.out‚0 disk.sda3.out‚ 34.64453125$ memory_percent‚96.97036743164062 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda3.in‚0# disk.nvme0n1.out‚4884.943359375 network.sent‚ 2.800196e+06* disk./.usagePercent‚39.588959013562416 disk.sda2.in‚0 disk.rbd0.in‚0 cpu‚25.867541728086817 network.recv‚461760% disk.nvme0c0n1.out‚4884.943359375$ disk./.usageGB‚589.9040985107422! disk.nvme0c0n1.in‚ 1940.359375 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 34.64453125 disk.dm-0.in‚ 1940.359375 disk.dm-0.out‚4884.943359375# proc.memory.rssMB‚ 2691.04296875) proc.memory.percent‚16.42482280731201XQùÝ ‚(:’ µúìÒìâüì gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Š¤~¸Þ ‚(:­ µúìÒ½í”ï disk.sda1.in‚0 disk.dm-0.in‚ 1940.359375# proc.memory.rssMB‚ 2691.04296875! disk.nvme0c0n1.in‚ 1940.359375 cpu‚25.8660498837658 network.sent‚ 2.849888e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 34.9609375$ disk.nvme0n1.out‚4890.9892578125$ disk./.usageGB‚589.9045944213867 disk.sda.out‚ 34.9609375 disk.rbd1.out‚4.296875$ memory_percent‚96.96948528289795) proc.memory.percent‚16.42482280731201) disk./.usagePercent‚39.58899229454187 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.out‚ 34.9609375 disk.drbd2165.in‚0! disk.dm-0.out‚4890.9892578125 disk.sda2.out‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚4890.9892578125( proc.memory.availableMB‚ 496.51953125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 1940.359375 network.recv‚466435ö°§žß ‚(:“ ÄúìÒ¢½óí gpu.0.gpu‚65.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£H½»à ‚(:° ÄúìÒ阊ï$ memory_percent‚96.97010517120361 cpu‚25.86455193965561 disk.drbd2165.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚4895.19287109375 disk.sda2.in‚0 network.sent‚ 2.85491e+06 disk.dm-1.in‚0 disk.nvme0n1.in‚ 1940.359375) proc.memory.percent‚16.42482280731201 disk.sda1.out‚0# proc.memory.rssMB‚ 2691.04296875 disk.sda3.out‚ 35.3203125' disk.nvme0c0n1.out‚4895.19287109375 disk.sda.out‚ 35.3203125 disk.sda1.in‚0$ disk./.usageGB‚589.9046783447266 disk.sda3.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 1940.359375( proc.memory.availableMB‚ 496.41796875 disk.dm-1.out‚ 35.3203125 disk.drbd2165.out‚0 disk.sda.in‚0 network.recv‚469597) disk./.usagePercent‚39.58899792670763 disk.rbd0.out‚0% disk.nvme0n1.out‚4895.19287109375 disk.dm-0.in‚ 1940.359375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚4.296875êxr$2á - _wandbruntime‚1642  _runtime‚1642€œ­|Çâ Á _step‚152 & lr-AdamW/pg1‚0.0005736632074135133 & lr-AdamW/pg2‚0.0005736632074135133 trainer/global_step‚1899 % _timestamp‚1.7843643642646132e+09 _runtime‚1642.770792157ÅšÌã Æ trainer/global_step‚1899 % _timestamp‚1.7843643642646132e+09 _runtime‚1642.770792157 _step‚152 & lr-AdamW/pg1‚0.0005736632074135133 & lr-AdamW/pg2‚0.0005736632074135133˜váÙû2ä - _wandbruntime‚1642  _runtime‚1642ÁÛÚ'å ÔO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17547259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287512046845952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7403675 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288035092234230 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2020988924928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287512063885312 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17557236 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65723238792192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4072214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4071933 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288035449823480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚1899 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1997952873472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7403391 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3331461 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287512046845952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32534583 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2021081696256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17557236 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63702268479488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289509999719424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1995236593912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚1642.772098873 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63702249867264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65723350175744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ìej¢ß'æ ÙO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14987326 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17547259 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287512046845952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7403675 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚288035092234230 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2020988924928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚287512063885312 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289509999719424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17557236 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65723238792192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14987324 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4072214 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4071933 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288035449823480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚1899 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1997952873472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1997952873472 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14987326 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7403391 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3331461 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14987324 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287512046845952 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32534583 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2021081696256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17557236 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63702268479488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289509999719424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1995236593912 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚1642.772098873 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63702249867264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1998294212608 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65723350175744™Ò £2ç - _wandbruntime‚1643  _runtime‚1643c­}¡è ÞM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286186723901440 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7407198 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17566302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287659287875584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 _runtime‚1643.401832862 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _step‚154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14994958 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7407468 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843643648955464e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3333206 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288182971549186 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289658611761664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288182613959824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65757688196608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2022098140672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289658253376000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287659304914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4074262 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1999306846720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286186706862080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101Ráé&é ãM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14994956 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1995907097744 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286186723901440 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7407198 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17566302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287659287875584 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 _runtime‚1643.401832862 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 _step‚154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14994958 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7407468 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2022009570816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 % _timestamp‚1.7843643648955464e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3333206 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288182971549186 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚289658611761664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288182613959824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚65757688196608 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2022098140672 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289658253376000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287659304914944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚65757795378688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4074262 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4073995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1999306846720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286186706862080 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32561260 Fä“ï ‚(:ˆ âúìÒåβí gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷µ(í¿ð ‚(:´ âúìÒ¸íï) disk./.usagePercent‚39.58891062813844# disk./.usageGB‚589.903377532959% disk.nvme0n1.out‚4906.38330078125 disk.rbd1.out‚4.375 disk.dm-1.in‚0 disk.dm-1.out‚ 35.80859375 disk.sda2.out‚0$ memory_percent‚96.97184562683105 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 35.80859375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 1997.5859375' disk.nvme0c0n1.out‚4906.38330078125 disk.sda3.out‚ 35.80859375' proc.memory.availableMB‚ 496.1328125 network.recv‚477604 disk.nvme0n1.in‚ 1997.5859375 disk.dm-0.in‚ 1997.5859375 disk.rbd1.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2691.05078125 cpu‚25.861457792265853" disk.dm-0.out‚4906.38330078125 disk.sda.in‚0* proc.memory.percent‚16.424870491027832 proc.cpu.threads‚17 network.sent‚ 2.922458e+06 disk.rbd0.in‚0ÐË-Øñ ‚(:’ ñúìÒ‰ÂÓï gpu.0.gpu‚25.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚209.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.964600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>PVi²ò ‚(:§ ñúìÒòÖÑò network.recv‚481171 disk.drbd2165.in‚0 disk.rbd1.in‚0$ memory_percent‚96.97034358978271 disk.sda3.out‚ 36.0546875 disk.rbd0.out‚0 disk.rbd1.out‚4.4375( proc.memory.availableMB‚ 496.37890625 disk.sda3.in‚0 disk.nvme0n1.in‚ 1997.5859375! disk.nvme0n1.out‚ 4913.2265625 disk.dm-0.in‚ 1997.5859375# proc.memory.rssMB‚ 2691.05078125) disk./.usagePercent‚39.58892701262064 disk.drbd2165.out‚0 disk.dm-0.out‚ 4913.2265625 network.sent‚ 2.929397e+06# disk./.usageGB‚589.903621673584 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 1997.5859375# disk.nvme0c0n1.out‚ 4913.2265625 disk.sda1.out‚0 cpu‚25.859949252427423 proc.cpu.threads‚17 disk.dm-1.out‚ 36.0546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 36.0546875 disk.sda1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.424870491027832áµÒð2ó - _wandbruntime‚1683  _runtime‚1683XÄÇô Á & lr-AdamW/pg1‚0.0005730991495586863 & lr-AdamW/pg2‚0.0005730991495586863 trainer/global_step‚1949 % _timestamp‚1.7843644052831395e+09 _runtime‚1683.789102124 _step‚156¬!ÇUÌõ Æ & lr-AdamW/pg1‚0.0005730991495586863 & lr-AdamW/pg2‚0.0005730991495586863 trainer/global_step‚1949 % _timestamp‚1.7843644052831395e+09 _runtime‚1683.789102124 _step‚156œ},y/2ö - _wandbruntime‚1683  _runtime‚1683E Ú'÷ ÔO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18001711 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15368924 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294874115366912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293365729783808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67445973206016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2072266800128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65373706405888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296922746702848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7594582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293365746823168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2045512530930 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294874115366912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4176151 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295411242314738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18011688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚1949 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2048648375296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15368924 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67445861822464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18001711 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2048648375296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2045853080820 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294874098327552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3418711 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15368926 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294874098327552}^þ]ø ÙO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296922746702848 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7594582 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3418708 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚293365746823168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2045512530930 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pooTm³‚ l.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294874115366912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2048989714432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4176151 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚295411242314738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18011688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚157 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚1949 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2048648375296 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15368924 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67445861822464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296923105081344 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18001711 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2048648375296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2045853080820 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294874098327552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3418711 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15368926 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294874098327552 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18001711 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15368924 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294874115366912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚293365729783808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67445973206016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2072266800128 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚65373706405888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296922746702848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2050002348544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297071000359424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297071358745088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67480420844032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18020754 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7598696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293512240455680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15376558 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295021356396544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295021339357184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67480313661952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18010763 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295021339357184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15376556 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049661002240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 $ _timestamp‚1.784364405912638e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚28­¤Íè&û âM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18020754 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7598696 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚293512240455680 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15376558 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚295021356396544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295021339357184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67480313661952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚295558764040332 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2049661002240 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18010763 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚295021339357184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297071358745088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2073285679616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15376556 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2049661002240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 $ _timestamp‚1.784364405912638e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18010763 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚65407135164416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2073197109760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15376558 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚65407116552192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3420453 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2050002348544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297071000359424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297071358745088 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7598433 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67480420844032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚295559121629694 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304ž4*42ü - _wandbruntime‚1684  _runtime‚16847¨¡îý ‚ 2loss/SRqqt2/assignment_loss‚0.6394522359767352 /loss/FRt1/detection_loss‚0.6141013335945004 1loss/SRqqt2/detection_loss‚0.7387748350960479 _step‚159 0loss/FRt1/assignment_loss‚2.4991190923680024 1loss/FBt1/assignment_loss‚0.09270967313406032 &loss/total_loss‚11.504274080672854 epoch‚4 /loss/FRt2/detection_loss‚1.4243738090222553 1loss/SRqqt1/detection_loss‚1.0524175821767536 0loss/FBt1/detection_loss‚0.40688593010574126 1loss/FBt2/assignment_loss‚0.06887146049011784 /loss/FBt2/detection_loss‚1.8026856748121125 trainer/global_step‚1949 % _timestamp‚1.7843644059143388e+09 _runtime‚1684.420029176 /loss/FRt2/assignment_loss‚1.450776455774589 2loss/SRqqt1/assignment_loss‚0.7141059981219374äñC þ ‡ 0loss/FRt1/assignment_loss‚2.4991190923680024 1loss/FBt1/assignment_loss‚0.09270967313406032 &loss/total_loss‚11.504274080672854 epoch‚4 /loss/FRt2/detection_loss‚1.4243738090222553 1loss/SRqqt1/detection_loss‚1.0524175821767536 0loss/FBt1/detection_loss‚0.40688593010574126 1loss/FBt2/assignment_loss‚0.06887146049011784 /loss/FBt2/detection_loss‚1.8026856748121125 trainer/global_step‚1949 % _timestamp‚1.7843644059143388e+09 _runtime‚1684.420029176 /loss/FRt2/assignment_loss‚1.450776455774589 2loss/SRqqt1/assignment_loss‚0.7141059981219374 2loss/SRqqt2/assignment_loss‚0.6394522359767352 /loss/FRt1/detection_loss‚0.6141013335945004 1loss/SRqqt2/detection_loss‚0.7387748350960479 _step‚159Ÿ‡Rnˆžÿ ‚(:“ €ûìÒ´¦ëí gpu.0.gpu‚64.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚266.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áxyc¼€ ‚(:± €ûìÒ‰Œæï disk.rbd0.out‚0 cpu‚25.85849760751416 network.recv‚484537 disk.sda1.out‚0 disk.dm-0.in‚ 1997.5859375 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.58894211706516 disk.sda.in‚0 disk.dm-1.out‚ 36.59765625 disk.sda2.in‚0# proc.memory.rssMB‚ 2691.05078125$ disk./.usageGB‚589.9038467407227 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚4916.7509765625' proc.memory.availableMB‚ 496.3984375* proc.memory.percent‚16.424870491027832& disk.nvme0c0n1.out‚4916.7509765625 disk.sda3.in‚0 disk.drbd2165.in‚0$ memory_percent‚96.97022438049316 disk.sda.out‚ 36.59765625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 1997.5859375! disk.dm-0.out‚4916.7509765625 network.sent‚ 2.93242e+06 disk.rbd1.out‚4.4375" disk.nvme0c0n1.in‚ 1997.5859375 disk.sda3.out‚ 36.59765625}ÞŸÀ“ ‚(:ˆ ûìÒ’Þ½î gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²|Á콂 ‚(:² ûìÒÍç‹ï cpu‚25.856861031006602 proc.cpu.threads‚17$ disk./.usageGB‚589.9036598205566 disk.sda3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 1997.5859375! disk.dm-0.out‚4922.3271484375 disk.sda2.in‚0& disk.nvme0c0n1.out‚4922.3271484375 disk.sda.out‚ 36.8359375 network.recv‚489099$ disk.nvme0n1.out‚4922.3271484375 disk.sda.in‚0$ memory_percent‚96.96755409240723* proc.memory.percent‚16.424870491027832 disk.drbd2165.out‚0 disk.sda3.out‚ 36.8359375 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 1997.5859375 disk.rbd1.in‚0) disk./.usagePercent‚39.58892957269598 disk.sda1.in‚0# proc.memory.rssMB‚ 2691.05078125 network.sent‚ 2.995456e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 36.8359375 disk.sda2.out‚0 disk.rbd1.out‚4.515625' proc.memory.availableMB‚ 496.8359375 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 1997.58593752DΙ2ƒ - _wandbruntime‚1724  _runtime‚1724~òd:„ 'ÂÄÁ trainer/global_step‚1999 % _timestamp‚1.7843644463202171e+09 _runtime‚1724.826190279 _step‚160 & lr-AdamW/pg1‚0.0005725169737320003 & lr-AdamW/pg2‚0.0005725169737320003¸øĽÌ… Æ trainer/global_step‚1999 % _timestamp‚1.7843644463202171e+09 _runtime‚1724.826190279 _step‚160 & lr-AdamW/pg1‚0.0005725169737320003 & lr-AdamW/pg2‚0.0005725169737320003 ®ÝeF2† - _wandbruntime‚1724  _runtime‚1724.×i¥Ú'‡ ÔO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304335852064768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15750526 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123422339072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302236166848512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18466140 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2096129017838 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4280354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4280080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18456163 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69168566671360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67045144332288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1724.827662669 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2099685216256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15750524 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2099343877120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123327470592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15750526 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302787749984496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚161 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304335493686272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304335493686272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚69168566671360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67045144332288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1724.827662669 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2099685216256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15750524 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67045125720064 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2099343877120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302236166848512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123327470592 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15750526 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302787749984496 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2099685216256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3505961 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚161 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304335493686272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 140399616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304335852064768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15750526 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123422339072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302236166848512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18466140 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2096129017838 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4280354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4280080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0¡Ç„õOž‰ ‚(:“ žûìÒ®–øì gpu.0.gpu‚41.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚182.871' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.574200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Ð‡šÃŠ ‚(:¸ žûìÒ׉îï disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 37.15234375 disk.dm-1.out‚ 37.15234375 proc.cpu.threads‚17 network.recv‚492385$ disk./.usageGB‚589.9046363830566 disk.dm-1.in‚0! disk.nvme0n1.in‚ 2060.08984375 disk.sda.out‚ 37.15234375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2060.08984375 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 2060.08984375* proc.memory.percent‚16.424870491027832) disk./.usagePercent‚39.58899511062475 disk.rbd1.in‚0# memory_percent‚96.9705581665039# proc.memory.rssMB‚ 2691.05078125 cpu‚25.855635707865165 disk.rbd1.out‚4.515625' disk.nvme0c0n1.out‚4926.12646484375 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚4926.12646484375" disk.dm-0.out‚4926.12646484375% proc.memory.availableMB‚ 496.34375 network.sent‚ 3.000388e+06 disk.sda2.in‚0 disk.rbd0.out‚0ü"I*2‹ - _wandbruntime‚1725  _runtime‚1725êÚ$4ä&Œ ÞM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15758156 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18465215 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300837774049280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15758158 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15758158 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚1725.453845821 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67078573090816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100697850368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302383390838784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124344266240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69202898744832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302934914120840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3507703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304484105728512 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4282158 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302935271710202 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438îàÕ2È ãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15758156 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18465215 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300837774049280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.smallÎ*!%_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15758158 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300837791088640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15758158 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚1725.453845821 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67078573090816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 140399616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2100697850368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302383390838784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7789861 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2100356504064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124344266240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1999 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69202898744832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚302934914120840 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34223371 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2097480621562 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34223371 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3507703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304484105728512 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4282158 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚302935271710202 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438¢`»âõ2Ž - _wandbruntime‚1725  _runtime‚1725áÝp ÿ 1loss/FBt2/assignment_loss‚0.07035166296922485 1loss/SRqqt1/detection_loss‚1.2295727067841933 /loss/FBt1/detection_loss‚0.6077112407547182 .loss/FBt2/detection_loss‚1.089493942736038 % _timestamp‚1.7843644469492834e+09 _runtime‚1725.454983816 _step‚163 1loss/FBt1/assignment_loss‚0.10687573000339594 /loss/FRt1/detection_loss‚0.5429122019742638 1loss/SRqqt2/detection_loss‚0.7368330230211209 %loss/total_loss‚11.38443347407441 0loss/FRt1/assignment_loss‚2.5402350486479284 /loss/FRt2/assignment_loss‚1.621534118420967 2loss/SRqqt2/assignment_loss‚0.6026252345713019 trainer/global_step‚1999 /loss/FRt2/detection_loss‚1.5082351806620726 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7280533835291862 KW®  „ 1loss/FBt1/assignment_loss‚0.10687573000339594 /loss/FRt1/detection_loss‚0.5429122019742638 1loss/SRqqt2/detection_loss‚0.7368330230211209 %loss/total_loss‚11.38443347407441 0loss/FRt1/assignment_loss‚2.5402350486479284 /loss/FRt2/assignment_loss‚1.621534118420967 2loss/SRqqt2/assignment_loss‚0.6026252345713019 trainer/global_step‚1999 /loss/FRt2/detection_loss‚1.5082351806620726 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7280533835291862 1loss/FBt2/assignment_loss‚0.07035166296922485 1loss/SRqqt1/detection_loss‚1.2295727067841933 /loss/FBt1/detection_loss‚0.6077112407547182 .loss/FBt2/detection_loss‚1.089493942736038 % _timestamp‚1.7843644469492834e+09 _runtime‚1725.454983816 _step‚163£¡ ]…‘‘ ‚(:† ­ûìÒâÇ®ï gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o—…¿’ ‚(:´ ­ûìÒÀŒÔð$ memory_percent‚96.97039127349854$ disk./.usageGB‚589.9046669006348 disk.drbd2165.out‚0 disk.dm-0.in‚ 2060.08984375' disk.nvme0c0n1.out‚4930.62353515625" disk.dm-0.out‚4930.62353515625 disk.sda3.out‚ 37.515625 network.sent‚ 3.065227e+06 network.recv‚496803 disk.rbd1.in‚0# proc.memory.rssMB‚ 2689.73046875 disk.rbd0.in‚0 disk.sda.out‚ 37.515625 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 496.37109375) disk./.usagePercent‚39.58899715868502 disk.drbd2165.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚4930.62353515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚4.578125! disk.nvme0n1.in‚ 2060.08984375# disk.nvme0c0n1.in‚ 2060.08984375 disk.sda1.in‚0 disk.dm-1.out‚ 37.515625( proc.memory.percent‚16.4168119430542 cpu‚25.887012106811177 proc.cpu.threads‚17ïVÌ2“ - _wandbruntime‚1742  _runtime‚1742¾}ù<°=” ª{ ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 'particle/f_score‚0.6075005531311035 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8289009928703308 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8842592239379883 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5966985821723938 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5666475296020508 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.663398802280426 +jet/accuracy_2_of_2‚0.11639543622732162 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922317266464233 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5640485882759094 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5990676283836365 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12415563315153122 /particle/accuracy_3_of_3‚0.9994933605194092 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9823261499404907 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11346173286437988 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9831293225288391 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08012579381465912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7679041624069214 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.634029746055603 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6796358823776245 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09899202734231949 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 particle/accuracy_2_of_3‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15588483214378357 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01450282335281372 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.45109808444976807 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08012579381465912 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009263480082154274 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7424870729446411 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19369308650493622 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08113525062799454 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15303723514080048 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.636951208114624 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7421637177467346 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2324819564819336 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6413617134094238 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5975133776664734 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 /particle/accuracy_3_of_4‚0.9916666746139526 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.5827385187149048 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1057119369506836 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3472222089767456 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970903992652893 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7251855134963989 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7251860499382019 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10093557089567184 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740741789340973 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.573221743106842 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5885916352272034 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.599445641040802 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7195389866828918 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4283851981163025 (particle/accuracy‚0.8383092284202576 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5990676283836365 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13415010273456573 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09580396115779877 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.423979252576828 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5129166841506958 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7679041624069214 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.09413833171129227 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.886993944644928 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.622329592704773 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5959315896034241 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34592103958129883 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34431976079940796 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 /particle/accuracy_4_of_4‚0.9425009489059448 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09907863289117813 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09038648009300232 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.41163480281829834 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9191066026687622 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9608606696128845 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 _step‚164 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740741789340973 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.34314045310020447 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024151470512151718 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6986168026924133 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941912889480591 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5906581282615662 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9081951975822449 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6912538409233093 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 epoch‚4 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11847376823425293 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 trainer/global_step‚1999 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09142644703388214 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15596583485603333 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005566111765801907 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.2606569528579712 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5083069801330566 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8474994897842407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9884859919548035 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10101723670959473 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.991167426109314 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08971673250198364 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34514373540878296 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6429934501647949 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5077797174453735 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951846599578857 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3445388376712799 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5839484333992004 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3124571144580841 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44513770937919617 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9811777472496033 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11083333194255829 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13106301426887512 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09195450693368912 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚10.956838273310748 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896191954612732 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6429350972175598 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6679725646972656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7843644644234104e+09 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6051398515701294 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9052391052246094 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850560426712036 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07094407826662064 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6903473734855652 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13091738522052765 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7004584074020386 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7125094532966614 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7127740383148193 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09226398915052414 +jet/accuracy_3_of_3‚0.05695031210780144 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955652952194214 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0176788829267025 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7226980924606323 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_2‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.663049578666687 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.990214467048645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6561036109924316 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3546222150325775 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10559163987636566 *jet/accuracy_2_of_4‚0.7372872233390808 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6272676587104797 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.061374466866254807 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_2‚0.7548173666000366 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7152683734893799 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937561750411987 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15423133969306946 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09195450693368912 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11655076593160629 particle/accuracy_2_of_4‚1 _runtime‚1742.930108573 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6424469947814941 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977500438690186 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970903992652893 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0030453659128397703 ,jet/accuracy_4_of_4‚0.007130124606192112 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6423371434211731 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929687976837158 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6987043023109436 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07690078020095825 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9846875071525574 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10340683907270432 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24528385698795319 particle/accuracy_3_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9769719243049622 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +jet/accuracy_3_of_4‚0.06248568370938301 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8278510570526123?ŒN-• ¯{ ,jet/accuracy_4_of_4‚0.007130124606192112 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6423371434211731 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9929687976837158 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6987043023109436 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07690078020095825 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9846875071525574 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10340683907270432 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24528385698795319 particle/accuracy_3_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9769719243049622 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 +jet/accuracy_3_of_4‚0.06248568370938301 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8278510570526123 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 'particle/f_score‚0.6075005531311035 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8289009928703308 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8842592239379883 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5966985821723938 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5666475296020508 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.663398802280426 +jet/accuracy_2_of_2‚0.11639543622732162 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922317266464233 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5640485882759094 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5990676283836365 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12415563315153122 /particle/accuracy_3_of_3‚0.9994933605194092 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9823261499404907 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11346173286437988 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9831293225288391 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08012579381465912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7679041624069214 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.634029746055603 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6796358823776245 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09899202734231949 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3671875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9793651103973389 particle/accuracy_2_of_3‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15588483214378357 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01450282335281372 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.4375 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.65625 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.45109808444976807 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08012579381465912 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009263480082154274 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7424870729446411 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19369308650493622 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08113525062799454 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15303723514080048 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.636951¤÷„Pˆ,208114624 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7421637177467346 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2324819564819336 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6413617134094238 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5975133776664734 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9941778779029846 /particle/accuracy_3_of_4‚0.9916666746139526 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.28125 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/sensitivity‚0.5827385187149048 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1057119369506836 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3472222089767456 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970903992652893 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7251855134963989 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7251860499382019 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10093557089567184 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740741789340973 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.573221743106842 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5885916352272034 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.599445641040802 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7195389866828918 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4283851981163025 (particle/accuracy‚0.8383092284202576 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5990676283836365 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13415010273456573 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09580396115779877 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.423979252576828 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5129166841506958 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7679041624069214 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.09413833171129227 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.886993944644928 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.622329592704773 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5959315896034241 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34592103958129883 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34431976079940796 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 /particle/accuracy_4_of_4‚0.9425009489059448 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09907863289117813 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09038648009300232 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.41163480281829834 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9191066026687622 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9608606696128845 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 _step‚164 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740741789340973 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.34314045310020447 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.024151470512151718 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6986168026924133 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941912889480591 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5906581282615662 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9081951975822449 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.578125 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6912538409233093 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 epoch‚4 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11847376823425293 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 *"Purity/2FRt0SRqqt2FBt/event_purity‚0.5 trainer/global_step‚1999 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09142644703388214 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15596583485603333 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005566111765801907 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.2606569528579712 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5083069801330566 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8474994897842407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9884859919548035 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10101723670959473 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.991167426109314 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08971673250198364 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34514373540878296 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6429934501647949 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5077797174453735 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951846599578857 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3445388376712799 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5839484333992004 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3124571144580841 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44513770937919617 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9811777472496033 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11083333194255829 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13106301426887512 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09195450693368912 jet/accuracy_2_of_5‚1 *loss/val_total_loss‚10.956838273310748 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9896191954612732 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6429350972175598 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6679725646972656 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7843644644234104e+09 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6051398515701294 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9052391052246094 particle/accuracy_4_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9850560426712036 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07094407826662064 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6903473734855652 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13091738522052765 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7004584074020386 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7125094532966614 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7127740383148193 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09226398915052414 +jet/accuracy_3_of_3‚0.05695031210780144 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9955652952194214 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0176788829267025 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7226980924606323 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 particle/accuracy_1_of_2‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.663049578666687 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.990214467048645 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6561036109924316 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3546222150325775 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10559163987636566 *jet/accuracy_2_of_4‚0.7372872233390808 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6272676587104797 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.061374466866254807 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_2‚0.7548173666000366 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7152683734893799 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9937561750411987 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15423133969306946 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09195450693368912 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11655076593160629 particle/accuracy_2_of_4‚1 _runtime‚1742.930108573 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6424469947814941 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977500438690186 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970903992652893 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0030453659128397703¤_Ð5ž– ‚(:“ ¼ûìÒ‡ú÷í gpu.0.gpu‚60.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±9ێ× ‚(:¸ ¼ûìÒŽèÌï disk.drbd2165.out‚0# proc.memory.rssMB‚ 2689.73046875 network.sent‚ 3.104488e+06$ disk./.usageGB‚589.9057121276855 disk.rbd0.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 743.76171875 proc.cpu.threads‚17 network.recv‚501213 disk.dm-0.in‚ 2060.08984375 disk.sda2.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚5178.86279296875 disk.dm-1.in‚0 disk.dm-1.out‚ 37.9296875( proc.memory.percent‚16.4168119430542 disk.rbd0.out‚0% disk.nvme0n1.out‚5178.86279296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0* disk./.usagePercent‚39.589067304749406 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 2060.08984375 disk.sda3.out‚ 37.9296875" disk.dm-0.out‚5178.86279296875 disk.rbd1.out‚4.578125$ memory_percent‚95.46043872833252 cpu‚25.890243561834343 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 2060.08984375 disk.sda.out‚ 37.9296875âˆb2˜ ‚(:„ ËûìÒï gpu.0.gpu‚9.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øð½™ ‚(:² ËûìÒ¹¼¤ð$ memory_percent‚95.46005725860596 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.in‚0 network.sent‚ 3.161603e+06) disk./.usagePercent‚39.58900407088844$ disk./.usageGB‚589.9047698974609 disk.sda3.out‚ 38.2578125 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚4.65625# proc.memory.rssMB‚ 2689.73046875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 2060.08984375( proc.memory.percent‚16.4168119430542 cpu‚25.88863355579567 disk.dm-1.out‚ 38.2578125& disk.nvme0c0n1.out‚5428.7216796875 disk.sda1.out‚0! disk.nvme0n1.in‚ 2060.08984375$ disk.nvme0n1.out‚5428.7216796875 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 network.recv‚505516! disk.dm-0.out‚5428.7216796875 disk.sda2.out‚0( proc.memory.availableMB‚ 743.82421875 proc.cpu.threads‚17 disk.dm-0.in‚ 2060.08984375 disk.sda.out‚ 38.2578125kL¾¬“š ‚(:ˆ ÚûìÒÖÌð gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚270.654' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥v6º› ‚(:¯ ÚûìÒã ô disk.rbd1.out‚4.65625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2689.73046875 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 38.578125 network.recv‚508924$ disk./.usageGB‚589.9047660827637 disk.dm-0.in‚ 2148.01953125 disk.drbd2165.out‚0 disk.rbd0.out‚0 network.sent‚ 3.16864e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.out‚ 38.578125! disk.nvme0n1.in‚ 2148.01953125 disk.sda1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2148.01953125 disk.sda3.out‚ 38.578125$ memory_percent‚95.46000957489014( proc.memory.percent‚16.4168119430542 cpu‚25.886951727991487$ disk.nvme0n1.out‚5431.6728515625 disk.sda2.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 743.83203125 proc.cpu.threads‚17& disk.nvme0c0n1.out‚5431.6728515625) disk./.usagePercent‚39.58900381488091! disk.dm-0.out‚5431.6728515625ûßE“2œ - _wandbruntime‚1785  _runtime‚1785Bã´½Æ À & lr-AdamW/pg1‚0.0005719167173255211 & lr-AdamW/pg2‚0.0005719167173255211 trainer/global_step‚2049 $ _timestamp‚1.784364506886739e+09 _runtime‚1785.392735207 _step‚165Þ™%Ëž Å & lr-AdamW/pg1‚0.0005719167173255211 & lr-AdamW/pg2‚0.0005719167173255211 trainer/global_step‚2049 $ _timestamp‚1.784364506886739e+09 _runtime‚1785.392735207 _step‚165¥¨ÜÙ2Ÿ - _wandbruntime‚1785  _runtime‚1785ÚÚÇéÚ'  ÔO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2167772587008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3595728 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚70925211819520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2192736073216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312090283004318 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312089925415068 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚70925316911616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚311527767941120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚311527784980480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4414742 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16252790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313695898906624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2168113926144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2164805779870 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2168113926144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8010473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309925477224448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2164465229980 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8010210 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311527784980480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚166 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311527767941120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16252792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18990919 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68732475746304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18990919 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18980942 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16252792 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2167772587008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16252790 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18980942 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3595731 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚68732494358528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309925460185088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚1785.397813499 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35243711 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 % _timestamp‚1.7843645068878052e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35233732 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313695898906624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4414482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2192822553088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313695540528128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16252790 QIDeviceStÏ"R‹atsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313695898906624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2168113926144 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2164805779870 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2168113926144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8010473 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309925477224448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2164465229980 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8010210 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚311527784980480 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _step‚166 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚311527767941120 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16252792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18990919 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚68732475746304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169¦»"ã2¢ - _wandbruntime‚1786  _runtime‚1786Zr¨Kä&£ ÞM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313844152570368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310071987896320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8014306 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2193843304448 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3597473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311675026010112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2169126560256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2169126560256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4416830 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4416565 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚167 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18989994 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚1786.020958725 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311675026010112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311675008970752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0’Ïdé&¤ ãM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35260409 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16260422 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2165816833704 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18999985 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3597473 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312237447140662 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚311675026010112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2169126560256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚70959659239424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16260422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚311675008970752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2165476283702 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2169126560256 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4416830 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4416565 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚167 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2168785213952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18999985 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18989994 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8014038 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _runtime‚1786.020958725 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚311675026010112 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚311675008970752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35250416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚313844152570368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚310071987896320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚310071970856960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16260424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3597476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8014306 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚70959766421504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2193843304448 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16634390 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16634392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19445371 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321108645890048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318889819422720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70403932284928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319466075495576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2215081716888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218809427968 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8201061 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2215422266778 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3682981 SGDeviceR&“ ´StatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3682978 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4518080 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72648063496704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317251010818048 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2244131211776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319466433084826 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1826.245797985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317250993778688 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚170 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19435394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16634392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321108287511552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19445371 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72647958404608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70403913672704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19435394 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318889836462080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218468088832 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321108645890048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2244044731904 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318889819422720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218809427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8200792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088ÊWƒß'± ÙO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2218809427968 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8201061 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2215422266778 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3682981 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3682978 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4518080 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72648063496704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317251010818048 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321108287511552 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2244131211776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319466433084826 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1826.245797985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317250993778688 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _step‚170 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19435394 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16634392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321108287511552 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16634390 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19445371 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72647958404608 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70403913672704 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19435394 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318889836462080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2218468088832 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321108645890048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2244044731904 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318889819422720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2218809427968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318889836462080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8200792 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2218468088832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2099 % _timestamp‚1.7843645477360792e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16634390 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16634392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19445371 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚321108645890048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318889819422720 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70403932284928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319466075495576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2215081716888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169ªêJA…2² - _wandbruntime‚1827  _runtime‚1827¤ò¯ã&³ ÝM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70437361043456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2216092770610 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16642022 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19454437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319037077491712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219480715776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2245061847040 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19444446 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2216433320612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317397521489920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317397504450560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8204825 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3684726 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319613954810532 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321256899553792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16642024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319037060452352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4519834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321256541168128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19444446 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319613597221170 % _timestamp‚1.7843645485531733e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16642024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002@¤T5´ âM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚319037077491712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2219822062080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70437361043456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2216092770610 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16642022 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19454437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319037077491712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2219480715776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2245061847040 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19444446 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2216433320612 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36086468 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317397521489920 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317397504450560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8204825 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3684726 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚319613954810532 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8204557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321256899553792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16642024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚319037060452352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4519834 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321256541168128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19444446 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319613597221170 % _timestamp‚1.7843645485531733e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16642024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4621194 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚326251887943680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2269163590656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72075351599104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚174 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19889846 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74370480959488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266038753686 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17015990 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4620927 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74370598634496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324576527372288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17015992 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19889846 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2295129360384 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19899823 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326251870904320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36915815 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015990 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326842583165334 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326842225576084 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265698203796 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328521034494976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17015992 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚326251870904320 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36915815 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17015990 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326842583165334 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326842225576084 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2265698203796 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4621194 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8391155 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚326251887943680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2269163590656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚72075351599104 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _step‚174 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36905836 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2269163590656 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19889846 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74370480959488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2266038753686 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17015990 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4620927 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74370598634496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚324576527372288 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0®ü÷2Ä - _wandbruntime‚1868  _runtime‚1868Ú¥¿vœ Å ÞM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267049807520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17023622 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2296241107456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8395228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17023624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326989747301678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocatedšã} H‚326399128973312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326399128973312 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17023624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843645895476375e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17023622 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328669646537216 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74404928700928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1868.053874389 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚326399111933952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328669646537216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19908889 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2270176217600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19898898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4622988 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72108780357632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2270517563904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2270176217600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2149 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ê~ÆÔé&Æ ãM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 % _timestamp‚1.7843645895476375e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17023622 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8394961 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328669646537216 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4623252 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74404928700928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚328669288151552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1868.053874389 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚326399111933952 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74405040077312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328669646537216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19908889 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2270176217600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19898898 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4622988 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚72108780357632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324723055083520 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2270517563904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2270176217600 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2149 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324723038044160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2267049807520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17023622 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2296241107456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8395228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17023624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19898898 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326989747301678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚326399128973312 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚326399128973312 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17023624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474¯ˆf‚½2Ç - _wandbruntime‚1868  _runtime‚1868¨Ÿ”È € 2loss/SRqqt2/assignment_loss‚0.6236811817503332 /loss/FRt1/detection_loss‚0.5814447705936412 /loss/FBt2/detection_loss‚1.0194005513874194 % _timestamp‚1.7843645895492837e+09 _runtime‚1868.054997499 0loss/FBt2/assignment_loss‚0.0320219672924092 /loss/FRt2/detection_loss‚1.6321844583909426 trainer/global_step‚2149 0loss/FRt1/assignment_loss‚2.4084528345607352 1loss/SRqqt1/detection_loss‚1.2242221093466996 /loss/FBt1/detection_loss‚0.5269131771055982 epoch‚5 0loss/FBt1/assignment_loss‚0.1976815414807101 1loss/SRqqt2/detection_loss‚0.7356056727942798 &loss/total_loss‚11.002589102494756 _step‚176 0loss/FRt2/assignment_loss‚1.2611103985990797 2loss/SRqqt1/assignment_loss‚0.7598704391929092Qƒj É … 1loss/SRqqt1/detection_loss‚1.2242221093466996 /loss/FBt1/detection_loss‚0.5269131771055982 epoch‚5 0loss/FBt1/assignment_loss‚0.1976815414807101 1loss/SRqqt2/detection_loss‚0.7356056727942798 &loss/total_loss‚11.002589102494756 _step‚176 0loss/FRt2/assignment_loss‚1.2611103985990797 2loss/SRqqt1/assignment_loss‚0.7598704391929092 2loss/SRqqt2/assignment_loss‚0.6236811817503332 /loss/FRt1/detection_loss‚0.5814447705936412 /loss/FBt2/detection_loss‚1.0194005513874194 % _timestamp‚1.7843645895492837e+09 _runtime‚1868.054997499 0loss/FBt2/assignment_loss‚0.0320219672924092 /loss/FRt2/detection_loss‚1.6321844583909426 trainer/global_step‚2149 0loss/FRt1/assignment_loss‚2.4084528345607352°‘5þ¤žÊ ‚(:“ ´üìÒ›Ûï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X±Bò¿Ë ‚(:´ ´üìÒ˜˜üð disk.sda3.out‚ 40.4921875 disk.rbd1.in‚0 disk.rbd1.out‚4.859375$ disk./.usageGB‚589.9053840637207 disk.rbd0.in‚0" disk.dm-0.out‚5458.11279296875" disk.nvme0c0n1.in‚ 2264.2109375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.96519374847412 disk.nvme0n1.in‚ 2264.2109375 network.recv‚530543) disk./.usagePercent‚39.58904528810146 disk.sda3.in‚0( proc.memory.availableMB‚ 497.22265625 proc.cpu.threads‚17 disk.dm-0.in‚ 2264.2109375# proc.memory.rssMB‚ 2689.73046875( proc.memory.percent‚16.4168119430542 network.sent‚ 3.325114e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚5458.11279296875' disk.nvme0c0n1.out‚5458.11279296875 disk.dm-1.in‚0 disk.dm-1.out‚ 40.4921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 40.4921875 cpu‚25.877924805571244v±ÝZ“Ì ‚(:ˆ ÃüìÒ³ßâí gpu.0.gpu‚41.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚210.888' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.1776 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸáqºÍ ‚(:¯ ÃüìÒ†ÙÜñ" disk.dm-0.out‚5462.49072265625 disk.sda3.out‚ 40.84765625# proc.memory.rssMB‚ 2689.73046875 cpu‚25.876380422054762 network.recv‚535441 disk.sda.in‚0 disk.sda1.out‚0( proc.memory.percent‚16.4168119430542 network.sent‚ 3.375172e+06$ disk./.usageGB‚589.9052925109863 disk.rbd0.in‚0% disk.nvme0n1.out‚5462.49072265625 disk.dm-1.in‚0 disk.dm-1.out‚ 40.84765625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 2264.2109375) disk./.usagePercent‚39.58903914392064 disk.sda2.in‚0 disk.rbd1.out‚5 disk.rbd1.in‚0 disk.sda.out‚ 40.84765625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 2264.2109375 disk.sda3.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚5462.49072265625 disk.drbd2165.in‚0$ memory_percent‚96.96536064147949' proc.memory.availableMB‚ 497.1953125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2264.2109375EZl“Î ‚(:ˆ ÒüìÒÝ°¥í gpu.0.gpu‚86.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚267.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¿®Ï ‚(:£ ÒüìÒØñ disk.dm-0.in‚ 2341.609375 disk.sda2.out‚0 disk.rbd1.out‚5! proc.memory.rssMB‚ 2689.734375 network.sent‚ 3.382208e+06 disk.dm-1.out‚41.15625 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.58903863190557 disk.sda1.in‚0 cpu‚25.874880256542603 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚0$ memory_percent‚96.96686267852783 network.recv‚538872' disk.nvme0c0n1.out‚5465.34814453125 disk.dm-1.in‚0" disk.dm-0.out‚5465.34814453125 disk.sda2.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 2341.609375 disk.sda1.out‚0( proc.memory.availableMB‚ 496.94921875 disk.sda.out‚41.15625 disk.sda3.out‚41.15625) proc.memory.percent‚16.41683578491211 disk.sda3.in‚0$ disk./.usageGB‚589.9052848815918 disk.nvme0n1.in‚ 2341.609375% disk.nvme0n1.out‚5465.34814453125¹×Ú‰2Ð - _wandbruntime‚1908  _runtime‚1908n§ÇÑ Á _runtime‚1908.514976505 _step‚177 & lr-AdamW/pg1‚0.0005700078559522742 & lr-AdamW/pg2‚0.0005700078559522742 trainer/global_step‚2199 % _timestamp‚1.7843646300087693e+09¯…¬ÏÌÒ Æ & lr-AdamW/pg2‚0.0005700078559522742 trainer/global_step‚2199 % _timestamp‚1.7843646300087693e+09 _runtime‚1908.514976505 _step‚177 & lr-AdamW/pg1‚0.0005700078559522742±Í CÃ2Ó - _wandbruntime‚1908  _runtime‚1908²;Ë :#Ô ÔO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333613922385920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843646300103016e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335933781478400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20344298 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4723509 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20354275 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17397592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 trainer/global_step‚2199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4723773 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335934139856896 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20354275 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8580987 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17397590 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333613939425280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3857478 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17397590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2316655240594 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2319859092480 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17397592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334218733245842 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319859092480 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8581254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2320200431616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚‹ƒªÍ 76092904985088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚178 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2316314690704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76093016368640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169ëì;­ß'Õ ÙO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2320200431616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73746808137728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73746789525504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚331902060965888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2319859092480 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8581254 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334218375656592 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333613939425280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2320200431616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76092904985088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2346115459584 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _step‚178 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3857481 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2316314690704 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76093016368640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333613922385920 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843646300103016e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚335933781478400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20344298 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4723509 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20354275 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17397592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2346208230912 trainer/global_step‚2199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335933781478400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4723773 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335934139856896 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20354275 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8580987 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17397590 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333613939425280 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37751867 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3857478 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17397590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2316655240594 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2319859092480 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17397592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334218733245842 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336082035134976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333761163415552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76127348601856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17405222 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73780218284032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4725819 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2347225179136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17405224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336082393520640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4725548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2320871719424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20353350 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8585045 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2317325744426 _runtime‚1909.150359243 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332048588677120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332048571637760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚179 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20363341 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37768565 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17405222% é&Ø ãM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333761180454912 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17405224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336082393520640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8584771 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4725548 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2320871719424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73780236896256 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20353350 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8585045 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76127462075392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2317325744426 _runtime‚1909.150359243 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332048588677120 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚332048571637760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321213065728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚179 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20363341 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37768565 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17405222 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334365897382186 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2320871719424 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336082035134976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚333761163415552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76127348601856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17405222 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20363341 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73780218284032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4725819 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2347225179136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343346528461824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75418246064128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚2249 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2397267202048 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8771537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340975990906880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2370895933440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚341594525737100 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341594883326350 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343346886840320 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2370554594304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2397355779072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77815601843200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2367271727502 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2370895933440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚182 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3944731 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339227594559488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340975990906880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343346886840320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20808727 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8771265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77815494653952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339227611598848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17779190 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20798750 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17779192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20808727 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616[ø/Þ'å ØO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚474 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75418227451904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚1949.466788937 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚2249 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2397267202048 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8771537 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚340975990906880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2370895933440 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚341594525737100 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚341594883326350 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343346886840320 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2370554594304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2397355779072 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚77815601843200 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2367271727502 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2370895933440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _step‚182 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3944731 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚339227594559488 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340975990906880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4826537 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343346886840320 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340975973867520 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20808727 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8771265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚77815494653952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚339227611598848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17779190 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17779190 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20798750 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17779192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20808727 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚539 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚340975973867520 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2370554594304 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343346528461824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343346528461824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75418246064128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339374122270720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339374105231360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341123214897152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343495140504064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341123214897152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341123231936512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20817793 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341742405052056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17786822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3946473 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17786824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2398377330688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 236794_jþÕü 2231334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20817793 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343494782118400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2398286663680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17786824 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77849942874112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2371908567552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2371567221248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8775322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2368282781336 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843646715868678e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3946476 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343495140504064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2371908567552 R÷;è&ê âM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3946476 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343495140504064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2371908567552 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341123231936512 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75451656210432 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4828846 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4828571 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚539 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚339374122270720 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚339374105231360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚341123214897152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343495140504064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341123214897152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341123231936512 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20817793 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚341742405052056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17786822 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3946473 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17786824 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8775044 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2398377330688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2367942231334 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20817793 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343494782118400 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2398286663680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17786824 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77849942874112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2371908567552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2371567221248 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚474 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8775322 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2368282781336 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843646715868678e+09·õ‚As2ë - _wandbruntime‚1950  _runtime‚1950$ `lì  $ _timestamp‚1.784364671588754e+09 2loss/SRqqt2/assignment_loss‚0.6151377308856417 /loss/FRt1/detection_loss‚0.6529462791209573 0loss/FBt1/detection_loss‚0.46110777061699926 1loss/FBt2/assignment_loss‚0.11516722917981086 .loss/FRt2/detection_loss‚1.525313489261039 1loss/SRqqt1/detection_loss‚1.0845380896857628 /loss/FBt2/detection_loss‚0.8021462238191263 0loss/FRt1/assignment_loss‚2.3143580693698462 0loss/FRt2/assignment_loss‚1.1134297123977117 1loss/FBt1/assignment_loss‚0.03402256636166613 &loss/total_loss‚10.135993352735094 epoch‚5 trainer/global_step‚2249 _runtime‚1950.094463844 _step‚184 2loss/SRqqt1/assignment_loss‚0.7099128617926909 1loss/SRqqt2/detection_loss‚0.7079133302438414RTéð í † 2loss/SRqqt2/assignment_loss‚0.6151377308856417 /loss/FRt1/detection_loss‚0.6529462791209573 0loss/FBt1/detection_loss‚0.46110777061699926 1loss/FBt2/assignment_loss‚0.11516722917981086 .loss/FRt2/detection_loss‚1.525313489261039 1loss/SRqqt1/detection_loss‚1.0845380896857628 /loss/FBt2/detection_loss‚0.8021462238191263 0loss/FRt1/assignment_loss‚2.3143580693698462 0loss/FRt2/assignment_loss‚1.1134297123977117 1loss/FBt1/assignment_loss‚0.03402256636166613 &loss/total_loss‚10.135993352735094 epoch‚5 trainer/global_step‚2249 _runtime‚1950.094463844 _step‚184 2loss/SRqqt1/assignment_loss‚0.7099128617926909 1loss/SRqqt2/detection_loss‚0.7079133302438414 $ _timestamp‚1.784364671588754e+09¸¤ÿy“î ‚(:ˆ ŽýìÒ«í–ð gpu.0.gpu‚86.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚148.428' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.6856 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߯?­ï ‚(:¢ ŽýìÒ¸ðõò disk.dm-1.out‚ 42.265625 disk.rbd0.out‚0' proc.memory.availableMB‚ 497.4140625 cpu‚25.869447487495947$ disk.nvme0c0n1.out‚ 5484.97265625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚16.41683578491211 proc.cpu.threads‚17 disk.nvme0n1.in‚ 2356.90625 disk.rbd1.out‚5.140625 disk.sda.in‚0 network.recv‚554120 disk.dm-0.out‚ 5484.97265625 disk.sda3.out‚ 42.265625 disk.rbd0.in‚0$ memory_percent‚96.96402549743652$ disk./.usageGB‚589.9051704406738 disk.dm-0.in‚ 2356.765625 disk.rbd1.in‚0" disk.nvme0n1.out‚ 5484.97265625) disk./.usagePercent‚39.58903095167954! disk.nvme0c0n1.in‚ 2356.765625 disk.dm-1.in‚0! proc.memory.rssMB‚ 2689.734375 disk.sda.out‚ 42.265625 network.sent‚ 3.516909e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚0óX3ã“ð ‚(:ˆ ýìҚDŽí gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ€ò8¼ñ ‚(:± ýìÒËÁüî) disk./.usagePercent‚39.58896157363776$ disk./.usageGB‚589.9041366577148 disk.sda.out‚ 42.55859375! proc.memory.rssMB‚ 2689.734375 cpu‚25.869782452531748 disk.rbd0.out‚0 disk.sda1.out‚0 network.recv‚558390 disk.sda3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 42.55859375 disk.sda1.in‚0# memory_percent‚96.9642162322998 disk.drbd2165.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚5489.8076171875! disk.dm-0.out‚5489.8076171875 disk.rbd1.out‚5.203125 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 3.556215e+06 disk.nvme0n1.in‚ 2414.8203125" disk.nvme0c0n1.in‚ 2414.8203125& disk.nvme0c0n1.out‚5489.8076171875 disk.sda.in‚0 disk.sda3.out‚ 42.55859375 disk.dm-0.in‚ 2414.8203125 disk.sda2.out‚0' proc.memory.availableMB‚ 497.3828125) proc.memory.percent‚16.41683578491211 disk.sda2.in‚0 disk.rbd1.in‚0D^ÿ^2ò - _wandbruntime‚1990  _runtime‚1990HAÍ÷Çó Á & lr-AdamW/pg1‚0.0005686456164674957 & lr-AdamW/pg2‚0.0005686456164674957 trainer/global_step‚2299 % _timestamp‚1.7843647122587845e+09 _runtime‚1990.764872077 _step‚185o˜éSÌô Æ % _timestamp‚1.7843647122587845e+09 _runtime‚1990.764872077 _step‚185 & lr-AdamW/pg1‚0.0005686456164674957 & lr-AdamW/pg2‚0.0005686456164674957 trainer/global_step‚2299¹¿Ÿ•D2õ - _wandbruntime‚1990  _runtime‚1990E¨ÔÚ'ö ÔO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348338025349120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4929317 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348970675817608 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2448286638592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚350759275445248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21253202 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18160792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4031981 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2417547664520 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79538063400448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4929597 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348971033406858 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2421250096128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21263179 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346553145192448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18160790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚186 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18160792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348338042388480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21253202 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4031978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77089683990528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2417888214410 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚1990.765880943 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79537952016896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77089665378304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348338042388480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350759633823744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888Udð÷ ÙO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚348971033406858 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2421250096128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21263179 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2421591435264 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346553145192448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346553128153088 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21263179 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39423971 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18160790 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _step‚186 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18160792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348338042388480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21253202 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4031978 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚77089683990528 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2417888214410 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _runtime‚1990.765880943 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350759275445248 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79537952016896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350759633823744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77089665378304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2448379409920 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚2299 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348338042388480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350759633823744 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348338025349120 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4929317 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚348970675817608 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2448286638592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚350759275445248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21253202 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18160792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4031981 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2417547664520 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79538063400448 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4929597 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1991.394534694 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18168422 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4033723 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79572398543872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4931623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346699638824960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2422262723072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18168424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2422262723072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77123094136832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2449405559808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350907887487488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2422604069376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348485266378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21272245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348485283418112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4033726 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚187 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18168424 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2418899268244 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350907887487488 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346699655864320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2418899268244 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚350907887487488 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39440669 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39430676 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚1991.394534694 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18168422 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4033723 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79572398543872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4931623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346699638824960 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2422262723072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18168424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2422262723072 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77123094136832 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2449405559808 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350907887487488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2422604069376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79572518308864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348485266378752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21272245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348485283418112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4033726 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77123112749056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8965083 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚187 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18168424 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21707654 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2471945597952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4119228 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81260394561024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355700076830720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21707654 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4119231 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78761103304704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353878661746688 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21717631 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356347183487366 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚2349 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358172022428672 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355700093870080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18542390 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78761121916928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9151816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356346825898116 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2032.397106529 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18542392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358172380807168 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2468504701318 _step‚190 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355700093870080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81260508041728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376³±ÿ ß'‰ ÙO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚358172022428672 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚355700093870080 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18542390 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚78761121916928 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9151534 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9151816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356346825898116 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2032.397106529 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18542392 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚358172380807168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚358172380807168 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2468504701318 _step‚190 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚358172022428672 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚355700093870080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81260508041728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2472286937088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21717631 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2468164151428 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40250044 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚40260023 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21707654 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2471945597952 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4119228 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81260394561024 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚355700076830720 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21707654 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4119231 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚78761103304704 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚353878661746688 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21717631 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356347183487366 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 trainer/global_step‚2349 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¾Õçw2Š - _wandbruntime‚2033  _runtime‚2033xe®ä&‹ ÞM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2472958224896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500400324608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2473299571200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356494347623710 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354025189457920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2469175205150 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355847317860352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18550022 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78794532063232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358320276085248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500311754752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2469175205150 trainer/global_step‚2349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21716706 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚355847317860352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18550022 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78794532063232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚358320276085248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2500311754752 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9155306 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚540 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 N>DeviceStatswœŽîRMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4120973 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2472958224896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2500400324608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2473299571200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356494347623710 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚354025189457920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363062128312320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363723333567874 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚194 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365584769412096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550324744704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚ 2073.86606452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2522641099776 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126063616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2522982438912 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80432559843328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361204212379648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361204195340288 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18923990 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2522982438912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2522641099776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9341634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22162106 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363062145351680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107451392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2519121188226 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363062145351680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365585127790592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80432541231104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4206481 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22162106 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363062128312320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5134877`Cý8'› ØO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365584769412096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363062145351680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 107451392 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2519121188226 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363062145351680 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365585127790592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80432541231104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4206481 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22162106 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363062128312320 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5134877 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 trainer/global_step‚2399 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚363722975978624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784364795359939e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5135153 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363062128312320 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚363723333567874 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚194 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365584769412096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2550324744704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚ 2073.86606452 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2522641099776 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41096075 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 126063616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2522982438912 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80432559843328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚361204212379648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚361204195340288 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 143733248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18923990 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2550217293312 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2522982438912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2522641099776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9341634 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚ i)|¦269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22162106Âßs2œ - _wandbruntime‚2074  _runtime‚2074*DXìä& ÞM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83017325079552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2520132242060 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22181149 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363870855293580 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2399 $ _timestamp‚1.784364795993019e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363209369341952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363209386381312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83017205314560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363209386381312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365733023068672 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41102780 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18931622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4208226 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872€jë$é&ž ãM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚361350723051520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5136913 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2519791692058 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2523995073024 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22181149 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2399 $ _timestamp‚1.784364795993019e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363209369341952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22171158 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚363209386381312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83017205314560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚363209386381312 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365733023068672 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41102780 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18931622 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9345136 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚286 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4208226 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365733381454336 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83017325079552 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363870497704218 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2520132242060 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18931622 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2551235324928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22181149 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 143733248 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363870855293580 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37074965238571167 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6383152604103088 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11095413565635681 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962903261184692 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8351998925209045 )jet/accuracy_1_of_2‚0.767418384552002 particle/accuracy_1_of_3‚1 /particle/accuracy_3_of_4‚0.9911564588546753 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6261957883834839 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15837396681308746 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6309353113174438 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7317506074905396 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.2919825315475464 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7360876798629761 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13106156885623932 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9919941425323486 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9846026301383972 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0052596754394471645 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27111631631851196 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04317264258861542 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99183589220047 'particle/f_score‚0.6284076571464539 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009672158397734165 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12711146473884583 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13106156885623932 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13160870969295502 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6141288876533508 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11083333194255829 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5354166626930237 6validation_average_jet_accuracy‚0.3404930830001831 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4556555449962616 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7408616542816162 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02294405922293663 particle/accuracy_1_of_1‚1 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.639438807964325 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6859055161476135 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7509448528289795 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47743260860443115 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1368277668952942 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740741789340973 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6157355308532715 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959355592727661 % _timestamp‚1.7843648145039995e+09 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7430603504180908 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8643494844436646 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5170425176620483 *jet/accuracy_1_of_4‚0.9755287170410156 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5386357307434082 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_3_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7083636522293091 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7360723614692688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 trainer/global_step‚2399 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07700838148593903 6"Purity/*FRt1SRqqt1FBt/event_purity‚0.4476338326931 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.127912700176239 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14396251738071442 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6857922673225403 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41049498319625854 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9947303533554077 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9760649800300598 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09174932539463043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8519824743270874 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4097222089767456 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 +jet/accuracy_3_of_3‚0.06728847324848175 particle/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23122435808181763 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13009525835514069 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7877532243728638 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740741789340973 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚9.953872375661124 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11365135759115219 /particle/accuracy_3_of_3‚0.9995894432067871 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _step‚197 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.044667843729257584 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/*FRt2SRqqt0FBt/SRqɢŤPqt_purity‚0.702393114566803 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6750378012657166 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0446743406355381 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.788275957107544 particle/accuracy_2_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181996583938599 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9885882139205933 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01866995356976986 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6390171647071838 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0733184963464737 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.009931244887411594 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03023987077176571 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *particle/sensitivity‚0.614830732345581 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.639438807964325 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970341920852661 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17942631244659424 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6342926621437073 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0991431400179863 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09652470052242279 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11401143670082092 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8093304634094238 /particle/accuracy_4_of_4‚0.9599627256393433 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2034522444009781 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04495197534561157 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9933992028236389 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5860283970832825 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1282089650630951 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845070838928223 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.986477792263031 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7540439367294312 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5711166858673096 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04425365850329399 *jet/accuracy_2_of_4‚0.8064054846763611 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9933686852455139 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6523074507713318 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7525888681411743 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7405204772949219 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.810044527053833 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9117393493652344 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6965125203132629 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880324006080627 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6369659900665283 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7398337721824646 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2940863072872162 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5872494578361511 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012000097893178463 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11046648025512695 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7439532279968262 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.586490273475647 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10184544324874878 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892122149467468 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3738398253917694 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18281464278697968 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6627855896949768 *jet/accuracy_1_of_3‚0.9217667579650879 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1456468552350998 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 +particle/specificity‚0.9064934849739075 epoch‚5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9938772320747375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8351998925209045 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5372143983840942 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1258174329996109 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7380426526069641 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_4‚0.09123896062374115 particle/accuracy_1_of_4‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14437010884284973 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.13850417733192444 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6893162131309509 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8844462633132935 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6898144483566284Ô,=¦ ‰{ @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9885882139205933 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01866995356976986 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6390171647071838 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0733184963464737 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.009931244887411594 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03023987077176571 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *particle/sensitivity‚0.614830732345581 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.639438807964325 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9970341920852661 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17942631244659424 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6342926621437073 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0991431400179863 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09652470052242279 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11401143670082092 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8093304634094238 /particle/accuracy_4_of_4‚0.9599627256393433 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2034522444009781 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04495197534561157 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9933992028236389 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5860283970832825 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1282089650630951 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845070838928223 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.986477792263031 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7540439367294312 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5711166858673096 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04425365850329399 *jet/accuracy_2_of_4‚0.8064054846763611 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9933686852455139 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6523074507713318 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7525888681411743 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7405204772949219 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.810044527053833 particle/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9117393493652344 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6965125203132629 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9880324006080627 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6369659900665283 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7398337721824646 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2940863072872162 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5872494578361511 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012000097893178463 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11046648025512695 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7439532279968262 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.586490273475647 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10184544324874878 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892122149467468 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3738398253917694 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18281464278697968 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6627855896949768 *jet/accuracy_1_of_3‚0.9217667579650879 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1456468552350998 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 +particle/specificity‚0.9064934849739075 epoch‚5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9938772320747375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8148148059844971 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8351998925209045 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5372143983840942 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1258174329996109 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7380426526069641 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_4‚0.09123896062374115 particle/accuracy_1_of_4‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14437010884284973 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.13850417733192444 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6893162131309509 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8844462633132935 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6898144483566284 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04317264258861542 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9946557283401489 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3987998366355896 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚2093.010663411 (particle/accuracy‚0.8438584208488464 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.683201253414154 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6355845928192139 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959355592727661 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.568566620349884 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8148148059844971 particle/accuracy_5_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4811176061630249 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9906066656112671 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37074965238571167 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6383152604103088 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11095413565635681 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962903261184692 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8351998925209045 )jet/accuracy_1_of_2‚0.767418384552002 particle/accuracy_1_of_3‚1 /particle/accuracy_3_of_4‚0.9911564588546753 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6261957883834839 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15837396681308746 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6309353113174438 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7317506074905396 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.2919825315475464 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7360876798629761 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13106156885623932 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9919941425323486 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9846026301383972 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0052596754394471645 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27111631631851196 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04317264258861542 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99183589220047 'particle/f_score‚0.6284076571464539 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009672158397734165 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.12711146473884583 particle/accuracy_2_of_3‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13106156885623932 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13160870969295502 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6141288876533508 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11083333194255829 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5354166626930237 6validation_average_jet_accuracy‚0.3404930830001831 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4556555449962616 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7408616542816162 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02294405922293663 particle/accuracy_1_of_1‚1 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.639438807964325 +"Purity/2FRt0SRqqt1FBt/event_purity‚0.25 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6859055161476135 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7509448528289795 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47743260860443115 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1368277668952942 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740741789340973 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6157355308532715 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959355592727661 % _timestamp‚1.7843648145039995e+09 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7430603504180908 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8643494844436646 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5170425176620483 *jet/accuracy_1_of_4‚0.9755287170410156 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5386357307434082 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 jet/accuracy_3_of_5‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7083636522293091 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7360723614692688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 trainer/global_step‚2399 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07700838148593903 6"Purity/*FRt1SRqqt1FBt/event_purity‚0.4476338326931 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.127912700176239 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14396251738071442 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_4‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6857922673225403 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41049498319625854 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9947303533554077 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9760649800300598 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09174932539463043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8519824743270874 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4097222089767456 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 +jet/accuracy_3_of_3‚0.06728847324848175 particle/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23122435808181763 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13009525835514069 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7877532243728638 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740741789340973 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚9.953872375661124 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11365135759115219 /particle/accuracy_3_of_3‚0.9995894432067871 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _step‚197 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.044667843729257584 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.702393114566803 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6750378012657166 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0446743406355381 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.788275957107544 particle/accuracy_2_of_5‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181996583938599Åq«ç“§ ‚(:ˆ •þìÒØò»ï gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚278.227' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6454 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š±ð¶¶¨ ‚(:« •þìÒÀ—¶ñ disk.dm-1.out‚ 45.1171875 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.921666608305436 disk.nvme0n1.in‚ 2609.0625 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 743.84765625) proc.memory.percent‚16.62008762359619 network.recv‚589174$ disk./.usageGB‚589.9060173034668 disk.sda2.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2723.03515625 network.sent‚ 3.769729e+06 disk.dm-0.in‚ 2609.0625" disk.dm-0.out‚5768.43798828125 disk.rbd1.out‚5.546875 disk.sda3.in‚0) disk./.usagePercent‚39.58908778535214 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 2609.0625 disk.sda1.in‚0# memory_percent‚95.4599142074585' disk.nvme0c0n1.out‚5768.43798828125 disk.sda.out‚ 45.1171875 disk.sda3.out‚ 45.1171875 proc.cpu.threads‚17 disk.drbd2165.out‚0% disk.nvme0n1.out‚5768.43798828125¢¨.¥ž© ‚(:“ ¤þìÒìã”í gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚229.484' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.896800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:q‘‘²ª ‚(:§ ¤þìÒ¶ã¡ï disk.rbd1.out‚5.546875) proc.memory.percent‚16.62008762359619 disk.dm-0.in‚ 2609.0625 disk.nvme0c0n1.in‚ 2609.0625 disk.dm-1.in‚0 network.sent‚ 3.826949e+06 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2723.03515625 network.recv‚593413$ disk.nvme0n1.out‚6017.1630859375! disk.dm-0.out‚6017.1630859375 disk.sda2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚589.9050750732422 disk.nvme0n1.in‚ 2609.0625 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 45.7734375 disk.sda1.out‚0* disk./.usagePercent‚39.589024551491185 disk.rbd0.out‚0$ memory_percent‚95.45955657958984% proc.memory.availableMB‚ 743.90625 disk.drbd2165.in‚0 disk.sda2.out‚0& disk.nvme0c0n1.out‚6017.1630859375 disk.sda3.out‚ 45.7734375 cpu‚25.924490318323603 disk.dm-1.out‚ 45.7734375“ÂáÝž« ‚(:“ ³þìÒÁ©î gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚241.365' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.272999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b&—¶¬ ‚(:« ³þìÒïצï) disk./.usagePercent‚39.58901635925009 disk.rbd0.out‚0 cpu‚25.92745142899852 disk.dm-1.out‚ 46.23046875 disk.drbd2165.out‚0 disk.rbd0.in‚0$ memory_percent‚95.46120166778564( proc.memory.availableMB‚ 743.63671875 disk.rbd1.out‚5.625' disk.nvme0c0n1.out‚6023.36376953125 disk.drbd2165.in‚0" disk.dm-0.out‚6023.36376953125 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚596765 disk.sda3.out‚ 46.23046875 disk.dm-0.in‚ 2609.0625 disk.sda2.in‚0$ disk./.usageGB‚589.9049530029297 disk.sda1.out‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚6023.36376953125 disk.rbd1.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 2609.0625 disk.sda.in‚0 disk.sda.out‚ 46.23046875 disk.nvme0n1.in‚ 2609.0625# proc.memory.rssMB‚ 2723.03515625) proc.memory.percent‚16.62008762359619 proc.cpu.threads‚17 network.sent‚ 3.832015e+06=€‰Ê2­ - _wandbruntime‚2136  _runtime‚21360ÚëÆ® À & lr-AdamW/pg1‚0.0005664686342936555 & lr-AdamW/pg2‚0.0005664686342936555 trainer/global_step‚2449 $ _timestamp‚1.784364857495805e+09 _runtime‚2136.001772693 _step‚198u߯˯ Å & lr-AdamW/pg1‚0.0005664686342936555 & lr-AdamW/pg2‚0.0005664686342936555 trainer/global_step‚2449 $ _timestamp‚1.784364857495805e+09 _runtime‚2136.001772693 _step‚198ƨz,2° - _wandbruntime‚2136  _runtime‚2136=jûUä± ÔO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372353763483648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22686885 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373025866587696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2619522658304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374945174632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2591411148800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2136.006752406 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84739325338624 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5272154 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚199 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2591069809664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372353746444288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2619434081280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374945174632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372353746444288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372353763483648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82119891257344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2591069809664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9568405 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373025508998446 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843648574968514e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22686885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370438392147968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22696862 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19426256 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2587457400368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5271893 trainer/global_step‚2449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19426256òqˆß'² ÙO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19426256 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9568141 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2587457400368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5271893 trainer/global_step‚2449 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19426256 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372353763483648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22686885 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373025866587696 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2619522658304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19426258 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚374945174632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2591411148800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84739432527872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2136.006752406 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4296248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84739325338624 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5272154 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2587116850478 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 _step‚199 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2591069809664 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372353746444288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2619434081280 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374945174632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372353746444288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374944816253952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372353763483648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82119891257344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2591069809664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9568405 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373025508998446 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 % _timestamp‚1.7843648574968514e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22686885 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚370438392147968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22696862Ƕßãf2³ - _wandbruntime‚2136  _runtime‚2136¯‡Qúã&´ ÝM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚2449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5274137 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372500987473920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373173388313402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚2136.634965641 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2592082436608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4297996 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433890 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19433888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84773885251072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19433888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370584902819840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82153338628096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370584919859200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375093428296192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22695937 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372500987473920 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2592082436608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2620451761664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0F¹wó µ âM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2588127904200 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚2449 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5274137 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372500987473920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚373173388313402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2592423782912 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚2136.634965641 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2592082436608 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9571865 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4297996 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19433890 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19433888 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚84773885251072 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19433888 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚370584902819840 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82153338628096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚370584919859200 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚375093428296192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22695937 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372500987473920 SEDeviceStatsMonitor.on_train_batch_end/active€ßÀÄõ_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚375093428296192 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2592082436608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2620451761664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2588468454202 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19807856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379715797925888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379715814965248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83791347795968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9759463 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382357921615872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19807858 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚2177.693883904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5375965 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638073887276 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2670703881728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2670784070144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2642106650624 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23151314 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83791329183744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843648991847186e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2641765311488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2641765311488 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382357921615872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77g¯TÞ'Ä ØO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19807858 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19807856 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19807856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379715797925888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379715814965248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83791347795968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9759463 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382357921615872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23151314 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19807858 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379715814965248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4383498 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚382357563237376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 _runtime‚2177.693883904 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5375965 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2638073887276 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86462033065472 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2670703881728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2670784070144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2642106650624 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23151314 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83791329183744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382357563237376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5376233 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 % _timestamp‚1.7843648991847186e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23141337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2641765311488 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2641765311488 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9759734 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2499 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚380401659078954 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23141337 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2642106650624 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚382357921615872 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380549538393910 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86496480528384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_bD_RSatch_end/inactive_split.large_pool.freed‚4385243 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379863038955520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2178.318117994 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19815490 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380549180804548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2639084941110 % _timestamp‚1.7843648998119874e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4385246 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚204 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2638744391108¸LM÷é&Ç ãM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2642777938432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2643119284736 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9763315 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚380549538393910 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚86496480528384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4385243 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379863038955520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9763574 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379863038955520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2178.318117994 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23160380 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83824757942272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2643119284736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19815490 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86496585613312 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚380549180804548 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2639084941110 % _timestamp‚1.7843648998119874e+09 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4385246 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚204 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2638744391108 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23150389 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382506175279616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚379863055994880 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2692802152448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9950977 _runtime‚2219.133273343 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387077866446848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4470748 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480487 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2722132530688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23605766 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20189456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4470751 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387077849407488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2692802152448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387077866446848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385089459335168 _step‚207 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23595789 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0x=?­Ô ÙO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2688349824294 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2692802152448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9950977 _runtime‚2219.133273343 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43785245 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389770668599296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚387077866446848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389770310220800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4470748 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88185002635776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480487 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85462767110144 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2722132530688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23605766 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20189456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2688690374184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4470751 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387077849407488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2692802152448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20189456 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387077866446848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚385089459335168 _step‚207 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23595789 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2692460813312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚387077849407488 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88184899640832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387777809159462 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2692460813312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5480229 trainer/global_step‚2549 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20189458 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚389770668599296Ï„é_2Õ - _wandbruntime‚2219  _runtime‚2219©9»Mã&Ö ÝM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387225090437120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20197090 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387925330885056 _runtime‚2219.768452204 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20197088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23614832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387225090437120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20197088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4472493 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88219351666688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385235970007040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2723240173568 trainer/global_step‚2549 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23604841 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389918922263040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2689360878016 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4472496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20197090 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23614832 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387225107476480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2693814786560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88219454654464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85496195868672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23604841 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2689701428018 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ƒ€þè&× âM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387225090437120 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20197090 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387925330885056 _runtime‚2219.768452204 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20197088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23614832 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85496214480896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387925688474418 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482312 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚385235987046400 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9954805 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387225090437120 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20197088 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4472493 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚88219351666688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚385235970007040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2723240173568 trainer/global_step‚2549 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23604841 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389918922263040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2689360878016 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4472496 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20197090 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23614832 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387225107476480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2693814786560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88219454654464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85496195868672 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23604841 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2689701428018 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43811922 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ði2Ø - _wandbruntime‚2219  _runtime‚2219d¡šVÙ ý 0loss/FRt2/assignment_loss‚1.1034423836727731 2loss/SRqqt1/assignment_loss‚0.7533836739519274 2loss/SRqqt2/assignment_loss‚0.6025437818745625 1loss/SRqqt2/detection_loss‚0.6832837932542782 /loss/FRt1/assignment_loss‚2.212541564006666 1loss/FBt1/assignment_loss‚0.11299528125667022 /loss/FBt2/assignment_loss‚0.562688482975938 &loss/total_loss‚10.477531253636144 epoch‚6 0loss/SRqqt1/detection_loss‚1.105307502978777 /loss/FBt1/detection_loss‚0.5197203915689949 /loss/FBt2/detection_loss‚0.7507738374988548 trainer/global_step‚2549 _runtime‚2219.769550216 /loss/FRt1/detection_loss‚0.5943139911019858 /loss/FRt2/detection_loss‚1.4765365694947157 $ _timestamp‚1.784364941263868e+09 _step‚209ÅPÚ ‚ /loss/FRt1/assignment_loss‚2.212541564006666 1loss/FBt1/assignment_loss‚0.11299528125667022 /loss/FBt2/assignment_loss‚0.562688482975938 &loss/total_loss‚10.477531253636144 epoch‚6 0loss/SRqqt1/detection_loss‚1.105307502978777 /loss/FBt1/detection_loss‚0.5197203915689949 /loss/FBt2/detection_loss‚0.7507738374988548 trainer/global_step‚2549 _runtime‚2219.769550216 /loss/FRt1/detection_loss‚0.5943139911019858 /loss/FRt2/detection_loss‚1.4765365694947157 $ _timestamp‚1.784364941263868e+09 _step‚209 0loss/FRt2/assignment_loss‚1.1034423836727731 2loss/SRqqt1/assignment_loss‚0.7533836739519274 2loss/SRqqt2/assignment_loss‚0.6025437818745625 1loss/SRqqt2/detection_loss‚0.6832837932542782ÑI=YžÛ ‚(:“ ÿìÒê­ží gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÉaÎÄÜ ‚(:¹ ÿìÒ‹Üéï disk.rbd1.out‚5.953125 disk.dm-1.in‚0( proc.memory.availableMB‚ 496.95703125) disk./.usagePercent‚39.58906423265899 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 2804.64453125 network.sent‚ 3.977278e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.944157421303018" disk.dm-0.out‚6049.92431640625 disk.sda1.out‚0" proc.memory.rssMB‚ 2723.0390625 disk.sda2.in‚0 disk.sda3.out‚ 48.30078125 disk.sda3.in‚0$ memory_percent‚96.96681499481201 network.recv‚618919 disk.dm-0.in‚ 2804.64453125 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 48.30078125( proc.memory.percent‚16.6201114654541$ disk./.usageGB‚589.9056663513184 disk.drbd2165.in‚0% disk.nvme0n1.out‚6049.92431640625 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 2804.64453125' disk.nvme0c0n1.out‚6049.92431640625 disk.sda1.in‚0 disk.dm-1.out‚ 48.30078125 proc.cpu.threads‚17C£ÀÇ“Ý ‚(:ˆ œÿìҞĎï gpu.0.gpu‚92.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QT{^¹Þ ‚(:® œÿìÒÒÉÑð disk.dm-0.in‚ 2804.64453125% proc.memory.availableMB‚ 497.15625 network.recv‚623352 disk.sda2.out‚0" proc.memory.rssMB‚ 2723.0390625 proc.cpu.threads‚17$ disk./.usageGB‚589.9057350158691 disk.sda2.in‚0 disk.rbd1.out‚5.953125 disk.sda.out‚ 48.640625 disk.sda3.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚6054.04541015625# disk.nvme0c0n1.in‚ 2804.64453125' disk.nvme0c0n1.out‚6054.04541015625 cpu‚25.946942549351128 network.sent‚ 4.042142e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 48.640625 disk.rbd1.in‚0! disk.nvme0n1.in‚ 2804.64453125" disk.dm-0.out‚6054.04541015625( disk./.usagePercent‚39.5890688407946 disk.drbd2165.out‚0# memory_percent‚96.9655990600586( proc.memory.percent‚16.6201114654541 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 48.640625 disk.drbd2165.in‚0WW»ì’ß ‚(:ˆ «ÿìÒïæ¾ï gpu.0.gpu‚44.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚Wçl0ˆ‘Àà ‚(:µ «ÿìÒ½·‰ñ disk.sda1.in‚0 disk.rbd1.out‚6.03125# disk.nvme0c0n1.in‚ 2804.64453125 disk.dm-1.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 497.41015625 disk.sda.in‚0 disk.dm-1.out‚ 49.0703125$ memory_percent‚96.96404933929443 cpu‚25.950249600860634 disk.sda2.out‚0 disk.sda.out‚ 49.0703125 disk.sda1.out‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚6062.05517578125 network.sent‚ 4.045106e+06! disk.nvme0n1.in‚ 2804.64453125( proc.memory.percent‚16.6201114654541 proc.cpu.threads‚17 disk.dm-0.in‚ 2804.64453125" disk.dm-0.out‚6062.05517578125 disk.sda3.out‚ 49.0703125$ disk./.usageGB‚589.9054641723633 disk.sda3.in‚0" proc.memory.rssMB‚ 2723.0390625) disk./.usagePercent‚39.58905066425968 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚626793 disk.rbd1.in‚0' disk.nvme0c0n1.out‚6062.05517578125xÛ’2á - _wandbruntime‚2260  _runtime‚2260ËþrÆâ À trainer/global_step‚2599 $ _timestamp‚1.784364982227794e+09 _runtime‚2260.733801628 _step‚210 & lr-AdamW/pg1‚0.0005641324299814468 & lr-AdamW/pg2‚0.0005641324299814468ãò«æËã Å _step‚210 & lr-AdamW/pg1‚0.0005641324299814468 & lr-AdamW/pg2‚0.0005641324299814468 trainer/global_step‚2599 $ _timestamp‚1.784364982227794e+09 _runtime‚2260.733801628ÒžápM2ä - _wandbruntime‚2260  _runtime‚2260‚¹ÊZÛ'å ÕO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚394439900889088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24050241 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24060218 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20571056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2599 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89907676495872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2738966311202 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394439917928448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89907573500928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2773452847104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395153959239970 $ _timestamp‚1.784364982228899e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24060218 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4558001 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4557998 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10142013 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397183057204224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2743497654272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397183415582720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2743497654272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20571056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2743156315136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚2260.735058952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÅGòÍà'æ ÚO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24050241 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2743156315136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚394439900889088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24050241 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24060218 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20571056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2599 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89907676495872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2738966311202 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚394439917928448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89907573500928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2739306861092 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2773452847104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚395153959239970 $ _timestamp‚1.784364982228899e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24060218 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4558001 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4557998 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44631276 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87134205036544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10142271 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397183057204224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10142013 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚397183057204224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2743497654272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚392414992928768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20571058 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5584015 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397183415582720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2743497654272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87134223648768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20571056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2743156315136 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚2260.735058952 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44621297 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89942131938816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4559746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20578690 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24069284 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5586273 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395301480965564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2744510288384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚2261.365338226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20578688 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394587158958080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20578690 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774479531520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394587141918720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2744168942080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87167652407296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397331669246464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2744168942080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2599 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10146019 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20578688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24059293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚397331669246464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24069284 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392561520640000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2744510288384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 % _timestamp‚1.7843649828592043e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397331310860800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2740317914926 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4559743 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394587158958080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ðY4*é åM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚2599 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10146019 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20578688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24059293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 T@DeviceStatsMonitor.onµÊ …Á#_train_batch_end/active_bytes.all.allocated‚397331669246464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24069284 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚392561520640000 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2744510288384 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 % _timestamp‚1.7843649828592043e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397331310860800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2740317914926 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4559743 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394587158958080 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚395301838554926 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚89942131938816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4559746 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚44637981 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20578690 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24069284 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5586273 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774393058816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚395301480965564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2744510288384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚2261.365338226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20578688 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394587158958080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20578690 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5586020 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774479531520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394587141918720 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2744168942080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87167652407296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397331669246464 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2744168942080 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚87167633795072 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10145763Ô¸iÅk2ê - _wandbruntime‚2261  _runtime‚2261^ùusë ‚ 1loss/SRqqt2/detection_loss‚0.6924398695088875 /loss/FBt2/detection_loss‚1.0400189694224133 _runtime‚2261.366382667 0loss/FRt2/assignment_loss‚1.1620181577546256 &loss/total_loss‚10.597735193915124 % _timestamp‚1.7843649828607018e+09 0loss/FRt1/assignment_loss‚2.2172746371995355 1loss/FBt1/assignment_loss‚0.10329495072699826 /loss/FBt1/detection_loss‚0.6491961108986288 2loss/SRqqt1/assignment_loss‚0.6728186408031684 2loss/SRqqt2/assignment_loss‚0.6002004956334601 /loss/FRt1/detection_loss‚0.6213407116232305 epoch‚6 trainer/global_step‚2599 _step‚213 1loss/FBt2/assignment_loss‚0.18007334049778517 /loss/FRt2/detection_loss‚1.5416782272333394 1loss/SRqqt1/detection_loss‚1.1173810826130504Šø/u ì ‡ 1loss/SRqqt1/detection_loss‚1.1173810826130504 1loss/SRqqt2/detection_loss‚0.6924398695088875 /loss/FBt2/detection_loss‚1.0400189694224133 _runtime‚2261.366382667 0loss/FRt2/assignment_loss‚1.1620181577546256 &loss/total_loss‚10.597735193915124 % _timestamp‚1.7843649828607018e+09 0loss/FRt1/assignment_loss‚2.2172746371995355 1loss/FBt1/assignment_loss‚0.10329495072699826 /loss/FBt1/detection_loss‚0.6491961108986288 2loss/SRqqt1/assignment_loss‚0.6728186408031684 2loss/SRqqt2/assignment_loss‚0.6002004956334601 /loss/FRt1/detection_loss‚0.6213407116232305 epoch‚6 trainer/global_step‚2599 _step‚213 1loss/FBt2/assignment_loss‚0.18007334049778517 /loss/FRt2/detection_loss‚1.5416782272333394Õ}ü¸6‘í ‚(:† ºÿìÒ¸½ší gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔÁî ‚(:¶ ºÿìÒìîßï disk.rbd0.out‚0 network.sent‚ 4.050025e+06 disk.sda3.out‚ 49.48828125 disk.drbd2165.out‚0 cpu‚25.95311867750873 disk.sda.out‚ 49.48828125 disk.sda1.out‚0 disk.dm-0.in‚ 2804.64453125 proc.cpu.threads‚17 network.recv‚630035) disk./.usagePercent‚39.58908343322406 disk.sda1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 496.6640625' disk.nvme0c0n1.out‚6066.38818359375 disk.sda3.in‚0 disk.drbd2165.in‚0$ disk./.usageGB‚589.9059524536133 disk.rbd1.in‚0! disk.nvme0n1.in‚ 2804.64453125" disk.dm-0.out‚6066.38818359375 disk.sda2.in‚0" proc.memory.rssMB‚ 2723.0390625( proc.memory.percent‚16.6201114654541 disk.sda2.out‚0 disk.rbd1.out‚6.03125# disk.nvme0c0n1.in‚ 2804.64453125 disk.sda.in‚0 disk.dm-1.in‚0% disk.nvme0n1.out‚6066.38818359375 disk.dm-1.out‚ 49.48828125$ memory_percent‚96.96860313415527´fÂ^‘ï ‚(:† ÉÿìÒ‹éúî gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùG¯ð ‚(:¤ ÉÿìÒæ‹Àð disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚6.09375( proc.memory.percent‚16.6201114654541) disk./.usagePercent‚39.58907396094529 disk.rbd0.out‚0! disk.dm-0.out‚6071.9169921875$ memory_percent‚96.96741104125977 disk.rbd0.in‚0 disk.rbd1.in‚0& proc.memory.availableMB‚ 496.859375 network.sent‚ 4.114799e+06 disk.sda3.out‚ 49.7578125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda.in‚0 disk.nvme0n1.in‚ 2863.3125 disk.sda.out‚ 49.7578125" proc.memory.rssMB‚ 2723.0390625 cpu‚25.955574662041524 disk.sda3.in‚0 disk.dm-1.out‚ 49.7578125$ disk.nvme0n1.out‚6071.9169921875 disk.nvme0c0n1.in‚ 2863.3125 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚634616$ disk./.usageGB‚589.9058113098145 disk.dm-0.in‚ 2863.3125 disk.sda2.in‚0& disk.nvme0c0n1.out‚6071.9169921875 proc.cpu.threads‚17{ÇCvñ ‚(:„ ØÿìÒ¿éí gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ÖFìò ‚(:¡ ØÿìÒáÏ€ï disk.sda2.out‚0 disk.sda.out‚ 50.078125 disk.drbd2165.out‚0 disk.rbd0.out‚0! disk.dm-0.out‚6074.7548828125 disk.sda3.out‚ 50.078125 disk.drbd2165.in‚0 disk.dm-0.in‚ 2863.3125 network.sent‚ 4.117763e+06) disk./.usagePercent‚39.58908215318639# disk./.usageGB‚589.905933380127& disk.nvme0c0n1.out‚6074.7548828125( proc.memory.availableMB‚ 497.00390625 disk.nvme0c0n1.in‚ 2863.3125 disk.sda3.in‚0( proc.memory.percent‚16.6201114654541 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 50.078125 disk.rbd0.in‚0 cpu‚25.95805205266304 network.recv‚637735 disk.nvme0n1.in‚ 2863.3125" proc.memory.rssMB‚ 2723.0390625 disk.rbd1.in‚0 disk.dm-1.in‚0$ memory_percent‚96.96652889251709 disk.rbd1.out‚6.09375 disk.sda.in‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚6074.7548828125 proc.cpu.threads‚17Gè]2ó - _wandbruntime‚2302  _runtime‚2302³Ó5ªÇô Á _runtime‚2302.355674897 _step‚214 & lr-AdamW/pg1‚0.0005633185432779665 & lr-AdamW/pg2‚0.0005633185432779665 trainer/global_step‚2649 % _timestamp‚1.7843650238497198e+09k±Ì{Ìõ Æ & lr-AdamW/pg1‚0.0005633185432779665 & lr-AdamW/pg2‚0.0005633185432779665 trainer/global_step‚2649 % _timestamp‚1.7843650238497198e+09 _runtime‚2302.355674897 _step‚214Ö“ÞC‚2ö - _wandbruntime‚2302  _runtime‚2302Z ª¼Ü'÷ ÖO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402530466909728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20952656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10333245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2824826462720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2793851816960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20952658 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚404595804187648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24504693 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5687994 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚88805661575168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402530109320478 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24514670 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399740543561728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚399740526522368 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20952656 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚404596162566144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4645248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88805642962944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2649 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚401801952370688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚215 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10332964 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91630488037888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2789923348000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚401801969410048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5687716 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401801969410048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2824742080000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843650238507447e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4645251 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24514670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4645248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚88805642962944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2649 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚401801952370688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚215 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10332964 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚91630488037888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2789923348000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚401801969410048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20952658 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5687716 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚401801969410048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2824742080000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7843650238507447e+09 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4645251 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚404595804187648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24514670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 I•æ&ú àM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚404744057844224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24523736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88839071721472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24513745 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402677631046072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2790934401834 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399887054233600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚2302.983012714 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401949193400320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843650244768264e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88839090333696 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4646996 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4646993 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5689808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10337073 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2825759602688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401949193400320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20960290 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91664942700544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2790593851832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2795205790208 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚88839090333696 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10336801 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4646996 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4646993 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5689808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24513745 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10337073 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2825759602688 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401949193400320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20960290 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚91664942700544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2790593851832 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24523736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚88839071721472 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24513745 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402677631046072 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2790934401834 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20960288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2825852366848 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399887054233600 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚2302.983012714 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401949193400320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5690077 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843650244768264e+09Øêëeæ2ü - _wandbruntime‚2302  _runtime‚2302s þ$ý  2loss/SRqqt2/assignment_loss‚0.5353006377511902 /loss/FRt1/detection_loss‚0.5918507347934785 &loss/total_loss‚10.143706133405052 % _timestamp‚1.7843650244781737e+09 0loss/FRt2/assignment_loss‚1.1209139656729814 1loss/FBt1/assignment_loss‚0.06576175201617405 3loss/FBt2/assignment_loss‚0.0025923084942673924 epoch‚6 2loss/SRqqt1/assignment_loss‚0.7471107508695164 /loss/FRt2/detection_loss‚1.4584957904672644 1loss/SRqqt2/detection_loss‚0.6222857904095231 /loss/FBt2/detection_loss‚1.1973308189189993 trainer/global_step‚2649 _runtime‚2302.983947368 /loss/SRqqt1/detection_loss‚1.06283147206088 /loss/FBt1/detection_loss‚0.5314611088843637 _step‚217 /loss/FRt1/assignment_loss‚2.207771003066414-€ þ † /loss/SRqqt1/detection_loss‚1.06283147206088 /loss/FBt1/detection_loss‚0.5314611088843637 _step‚217 /loss/FRt1/assignment_loss‚2.207771003066414 2loss/SRqqt2/assignment_loss‚0.5353006377511902 /loss/FRt1/detection_loss‚0.5918507347934785 &loss/total_loss‚10.143706133405052 % _timestamp‚1.7843650244781737e+09 0loss/FRt2/assignment_loss‚1.1209139656729814 1loss/FBt1/assignment_loss‚0.06576175201617405 3loss/FBt2/assignment_loss‚0.0025923084942673924 epoch‚6 2loss/SRqqt1/assignment_loss‚0.7471107508695164 /loss/FRt2/detection_loss‚1.4584957904672644 1loss/SRqqt2/detection_loss‚0.6222857904095231 /loss/FBt2/detection_loss‚1.1973308189189993 trainer/global_step‚2649 _runtime‚2302.983947368ÙG¸À“ÿ ‚(:ˆ çÿìÒ£¢ùì gpu.0.gpu‚59.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚210.597' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.1194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~bÖŠ°€ ‚(:¥ çÿìÒ¸§‡ï' proc.memory.availableMB‚ 496.7578125 network.recv‚640303$ disk./.usageGB‚589.9058799743652 disk.nvme0c0n1.in‚ 2863.3125 disk.dm-1.out‚ 50.4453125 proc.cpu.threads‚17 network.sent‚ 4.121758e+06 disk.rbd1.out‚6.171875 disk.sda1.in‚0 disk.sda3.out‚ 50.4453125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 2863.3125 disk.rbd1.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚6078.9404296875! disk.dm-0.out‚6078.9404296875 disk.sda2.out‚0) disk./.usagePercent‚39.58907856908091 disk.sda3.in‚0 disk.dm-1.in‚0$ memory_percent‚96.96803092956543" proc.memory.rssMB‚ 2723.0390625 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 2863.3125 disk.sda.out‚ 50.4453125 disk.sda1.out‚0( proc.memory.percent‚16.6201114654541 cpu‚25.96056888402516& disk.nvme0c0n1.out‚6078.9404296875 disk.drbd2165.out‚069´qž ‚(:“ öÿìÒé…Žï gpu.0.gpu‚96.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚217.657' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.531400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iµ?ȶ‚ ‚(:« öÿìÒ‹áÁð disk.dm-1.out‚ 50.75390625 disk.nvme0n1.in‚ 2863.3125% disk.nvme0n1.out‚6082.88427734375" disk.dm-0.out‚6082.88427734375) disk./.usagePercent‚39.58908189717885 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2723.0390625( proc.memory.percent‚16.6201114654541 disk.sda1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚6082.88427734375 disk.drbd2165.out‚0$ memory_percent‚96.96645736694336 network.sent‚ 4.186731e+06 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0& proc.memory.availableMB‚ 497.015625 disk.nvme0c0n1.in‚ 2863.3125 disk.rbd0.in‚0 disk.rbd1.out‚6.171875$ disk./.usageGB‚589.9059295654297 disk.dm-1.in‚0 disk.dm-0.in‚ 2863.3125 cpu‚25.962808384724724 proc.cpu.threads‚17 network.recv‚644441 disk.sda.out‚ 50.75390625 disk.sda3.in‚0 disk.sda3.out‚ 50.75390625…L¯‘ƒ ‚(:† …€íÒó£î gpu.0.gpu‚89.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ÀE±À„ ‚(:µ …€íÒºÖÅï disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 2935.5078125 disk.sda.out‚ 51.06640625( proc.memory.availableMB‚ 497.01953125 disk.dm-0.in‚ 2935.5078125 disk.dm-1.out‚ 51.06640625 disk.sda2.in‚0' disk.nvme0c0n1.out‚6087.90966796875" proc.memory.rssMB‚ 2723.0390625 proc.cpu.threads‚17" disk.dm-0.out‚6087.90966796875 network.sent‚ 4.189763e+06 disk.sda2.out‚0 disk.rbd1.out‚6.296875 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚589.9057960510254 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚6087.90966796875 disk.sda.in‚0$ memory_percent‚96.96643352508545 disk.nvme0n1.in‚ 2935.5078125 network.recv‚647818) disk./.usagePercent‚39.58907293691515 disk.sda3.out‚ 51.06640625 disk.rbd1.in‚0 cpu‚25.96148082442784( proc.memory.percent‚16.6201114654541müça2… - _wandbruntime‚2343  _runtime‚2343ë¹Å† ¿ trainer/global_step‚2699 $ _timestamp‚1.784365065037709e+09 _runtime‚ 2343.54380311 _step‚218 & lr-AdamW/pg1‚0.0005624871667926924 & lr-AdamW/pg2‚0.0005624871667926924 ¸[äʇ Ä & lr-AdamW/pg1‚0.0005624871667926924 & lr-AdamW/pg2‚0.0005624871667926924 trainer/global_step‚2699 $ _timestamp‚1.784365065037709e+09 _runtime‚ 2343.54380311 _step‚218Úï«2ˆ - _wandbruntime‚2343  _runtime‚2343t!.¼f ‰ ÔO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93353149074944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2844888657920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409164020891648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21334258 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21334256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93353043982848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2844547318784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2876049573376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412008551171072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4732501 VFDeviceStatsMÜAbõtonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5791525 _runtime‚ 2343.54872549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409164003852288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2875963093504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90477099501568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚407066077155328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2840199285018 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46293401 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409906616990236 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412008909549568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2840539834908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24969122 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21334258 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407066060115968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2844547318784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409906259400986 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2699 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90477080889344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2844888657920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24959145ÚÒKß'Š ÙO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2844888657920 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚412008909549568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24959145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93353149074944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24959145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2844888657920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚409164020891648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21334258 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21334256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93353043982848 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2844547318784 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2876049573376 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412008551171072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4732501 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5791525 _runtime‚ 2343.54872549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚409164003852288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2875963093504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412008551171072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90477099501568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚407066077155328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2840199285018 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46293401 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409906616990236 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚412008909549568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2840539834908 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24969122 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21334258 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407066060115968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2844547318784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚409906259400986 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚2699 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚90477080889344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983Û«L*2‹ - _wandbruntime‚2344  _runtime‚2344Š‰Ù~å&Œ ßM _runtime‚ 2344.17794336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409311244881920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2841550888742 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24968197 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412157163213312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845901292032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10527816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21341890 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412157163213312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21341890 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24968197 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410053781126580 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2877067443200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21341888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90510509647872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409311261921280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93387488521216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5793311 % _timestamp‚1.7843650656717596e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409311244881920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412156804827648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410054138715942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407212570787840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24978188 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4734243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845901292032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0Î9î s äM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24968197 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚412157163213312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845901292032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24978188 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10527816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21341890 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412157163213312 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21341890 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24968197 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410053781126580 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 LCDeviceStatsMV¸k]wonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2877067443200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21341888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90510509647872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚409311261921280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚409311261921280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93387488521216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5793311 % _timestamp‚1.7843650656717596e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚409311244881920 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412156804827648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410054138715942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407212570787840 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24978188 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4734243 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845901292032 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 2344.17794336 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10527554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚409311244881920 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2841550888742Ü7Õ¹õ2Ž - _wandbruntime‚2344  _runtime‚2344ó ´  ƒ 0loss/FRt1/assignment_loss‚2.3063754001417913 2loss/FBt1/assignment_loss‚0.049749863195854255 0loss/FBt1/detection_loss‚0.48749066528913654 % _timestamp‚1.7843650656734352e+09 _step‚221 /loss/FRt1/detection_loss‚0.5703596813138385 /loss/FRt2/detection_loss‚1.4449066283274625 &loss/total_loss‚10.087363309057766 _runtime‚ 2344.17911951 0loss/FRt2/assignment_loss‚1.1187809815809628 2loss/SRqqt1/assignment_loss‚0.8383490290731401 0loss/SRqqt1/detection_loss‚1.044889737109828 1loss/SRqqt2/detection_loss‚0.6334225423938407 2loss/SRqqt2/assignment_loss‚0.5617154127882803 2loss/FBt2/assignment_loss‚0.005146921726169458 /loss/FBt2/detection_loss‚1.0261764461174607 epoch‚6 trainer/global_step‚2699¿4a ˆ 0loss/FRt1/assignment_loss‚2.3063754001417913 2loss/FBt1/assignment_loss‚0.049749863195854255 0loss/FBt1/detection_loss‚0.48749066528913654 % _timestamp‚1.7843650656734352e+09 _step‚221 /loss/FRt1/detection_loss‚0.5703596813138385 /loss/FRt2/detection_loss‚1.4449066283274625 &loss/total_loss‚10.087363309057766 _runtime‚ 2344.17911951 0loss/FRt2/assignment_loss‚1.1187809815809628 2loss/SRqqt1/assignment_loss‚0.8383490290731401 0loss/SRqqt1/detection_loss‚1.044889737109828 1loss/SRqqt2/detection_loss‚0.6334225423938407 2loss/SRqqt2/assignment_loss‚0.5617154127882803 2loss/FBt2/assignment_loss‚0.005146921726169458 /loss/FBt2/detection_loss‚1.0261764461174607 epoch‚6 trainer/global_step‚2699ÝV§ên“‘ ‚(:ˆ ”€íÒʵÎï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R Ѿ’ ‚(:³ ”€íÒ¯·æñ) disk./.usagePercent‚39.58908752934461 disk.sda.out‚ 51.31640625 disk.dm-1.out‚ 51.31640625 disk.drbd2165.out‚0 cpu‚25.959991495377203 disk.rbd1.in‚0 disk.rbd1.out‚6.296875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2723.0390625 network.sent‚ 4.207722e+06 disk.nvme0n1.in‚ 2935.5078125$ disk.nvme0n1.out‚6091.3876953125( proc.memory.availableMB‚ 496.97265625$ disk./.usageGB‚589.9060134887695 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 2935.5078125 disk.sda1.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0& disk.nvme0c0n1.out‚6091.3876953125 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 51.31640625$ memory_percent‚96.96671962738037 disk.dm-0.in‚ 2935.5078125! disk.dm-0.out‚6091.3876953125 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 network.recv‚651205( proc.memory.percent‚16.6201114654541õ¤öÉ‘“ ‚(:† £€íÒ©äžï gpu.0.gpu‚60.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚251.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÂ9 ¹” ‚(:® £€íÒ¨ç…ò( proc.memory.percent‚16.6201114654541 disk.dm-0.in‚ 2935.5078125 disk.rbd1.out‚6.4375 disk.sda.out‚ 51.671875$ memory_percent‚96.96567058563232( proc.memory.availableMB‚ 497.14453125 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 2935.5078125 disk.sda.in‚0 network.recv‚655880 cpu‚25.958148666799858 proc.cpu.threads‚17) disk./.usagePercent‚39.58908394523913 disk.rbd0.in‚0 disk.sda1.in‚0 network.sent‚ 4.259489e+06$ disk./.usageGB‚589.9059600830078 disk.sda3.out‚ 51.671875 disk.drbd2165.out‚0" disk.dm-0.out‚6095.64013671875 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 2935.5078125 disk.sda1.out‚0 disk.dm-1.out‚ 51.671875 disk.sda2.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚6095.64013671875" proc.memory.rssMB‚ 2723.0390625 disk.drbd2165.in‚0% disk.nvme0n1.out‚6095.64013671875áQÆ2• - _wandbruntime‚2384  _runtime‚2384Ý„´Æ– À & lr-AdamW/pg1‚0.0005616383539233824 & lr-AdamW/pg2‚0.0005616383539233824 trainer/global_step‚2749 $ _timestamp‚1.784365106066207e+09 _runtime‚2384.572204074 _step‚222žs˜Ë— Å _step‚222 & lr-AdamW/pg1‚0.0005616383539233824 & lr-AdamW/pg2‚0.0005616383539233824 trainer/global_step‚2749 $ _timestamp‚1.784365106066207e+09 _runtime‚2384.572204074Þc¥Šc2˜ - _wandbruntime‚2384  _runtime‚2384nB•ÜÜ'™ ÖO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2895242820608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2927098980352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2890815771926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21715856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2895584159744 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4819751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25423574 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5893822 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417282409481494 % _timestamp‚1.7843651060672975e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10713830 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25423574 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927004111872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21715858 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21715856 _runtime‚2384.576746812 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21715858 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419421298154496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25413597 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416526055333888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417282767070744 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47129453 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419421656532992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5894079 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚223 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416526072373248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414391610748928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2927098980352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2890815771926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21715856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2895584159744 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4819751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25423574 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5893822 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚417282409481494 % _timestamp‚1.7843651060672975e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10713830 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25423574 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927004111872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚419421298154496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21715858 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21715856 _runtime‚2384.576746812 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚2749 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21715858 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚419421298154496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25413597 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416526055333888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚417282767070744 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47129453 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚419421656532992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5894079 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 _step‚223 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416526072373248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414391610748928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92181966186496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚417429931207088 M=DeviceStatsMonitor.on_trainæzwÄù_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2892167375650 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2896255447552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10717631 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416673313402880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2891826825648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25422649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419569910196736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2896255447552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21723488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419569910196736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚2385.200469649 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5895879 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 # _timestamp‚1.78436510669403e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4821496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640%`¹é&Ÿ ãM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2892167375650 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25422649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2896255447552 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10717631 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416673313402880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2891826825648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25422649 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚419569910196736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2896255447552 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2749 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21723488 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚419569910196736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5896135 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚2385.200469649 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2896596793856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5895879 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 # _timestamp‚1.78436510669403e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4821496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25432640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416673296363520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92181966186496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚417430288796450 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2896596793856 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4821493 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21723490 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚416673296363520 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10717372 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚417429931207088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169àk¡¼¢2  - _wandbruntime‚2385  _runtime‚2385ÿ¨í¿¡ ‚ /loss/FBt2/detection_loss‚1.1876387571227616 epoch‚6 2loss/SRqqt1/assignment_loss‚0.8033072439256062 /loss/FRt1/detection_loss‚0.5751644907723749 trainer/global_step‚2749 _step‚225 -loss/FRt2/detection_loss‚1.58058990362029 1loss/SRqqt2/detection_loss‚0.7360286964990853 % _timestamp‚1.7843651066954932e+09 0loss/FRt2/assignment_loss‚1.1605614987579551 2loss/FBt1/assignment_loss‚0.032659652957360216 2loss/FBt2/assignment_loss‚0.015240909705922474 1loss/SRqqt1/detection_loss‚0.9210470318446742 /loss/FBt1/detection_loss‚0.3914935518602633 &loss/total_loss‚10.234970370369098 _runtime‚2385.201415985 0loss/FRt1/assignment_loss‚2.2318469796180724 2loss/SRqqt2/assignment_loss‚0.5993916536847325ýî¿í ¢ ‡ 2loss/SRqqt1/assignment_loss‚0.8033072439256062 /loss/FRt1/detection_loss‚0.5751644907723749 trainer/global_step‚2749 _step‚225 -loss/FRt2/detection_loss‚1.58058990362029 1loss/SRqqt2/detection_loss‚0.7360286964990853 % _timestamp‚1.7843651066954932e+09 0loss/FRt2/assignment_loss‚1.1605614987579551 2loss/FBt1/assignment_loss‚0.032659652957360216 2loss/FBt2/assignment_loss‚0.015240909705922474 1loss/SRqqt1/detection_loss‚0.9210470318446742 /loss/FBt1/detection_loss‚0.3914935518602633 &loss/total_loss‚10.234970370369098 _runtime‚2385.201415985 0loss/FRt1/assignment_loss‚2.2318469796180724 2loss/SRqqt2/assignment_loss‚0.5993916536847325 /loss/FBt2/detection_loss‚1.1876387571227616 epoch‚6áú§¾“£ ‚(:ˆ Á€íÒÀ¸‹ð gpu.0.gpu‚85.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚112.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4838 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ÿ¸¤ ‚(:­ Á€íÒÿÁò% disk.nvme0n1.out‚6103.78662109375( proc.memory.percent‚16.6201114654541* disk./.usagePercent‚39.589097257630904 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 3005.25390625 disk.sda1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚6103.78662109375 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚96.96898460388184" proc.memory.rssMB‚ 2723.0390625 network.sent‚ 4.327413e+06 disk.rbd0.in‚0$ disk./.usageGB‚589.9061584472656 disk.dm-1.in‚0" disk.dm-0.out‚6103.78662109375 disk.sda2.out‚0 disk.rbd1.out‚6.5 network.recv‚663770 disk.sda.out‚ 52.328125 disk.sda3.in‚0 disk.dm-0.in‚ 3005.25390625 disk.dm-1.out‚ 52.328125 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 3005.25390625 disk.rbd1.in‚0' proc.memory.availableMB‚ 496.6015625 cpu‚25.95473817254151 disk.sda3.out‚ 52.328125X»å»“¥ ‚(:ˆ ЀíÒάøî gpu.0.gpu‚51.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+$1Ö¼¦ ‚(:± ЀíÒ—ûñ disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 3005.25390625( proc.memory.availableMB‚ 496.40234375 proc.cpu.threads‚17 disk.rbd0.in‚0$ memory_percent‚96.97020053863525) disk./.usagePercent‚39.58909674561584! disk.dm-0.out‚6106.3349609375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 52.58203125 cpu‚25.953410107283705 network.sent‚ 4.368797e+06$ disk./.usageGB‚589.9061508178711 disk.sda3.out‚ 52.58203125 disk.dm-1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚6106.3349609375 disk.rbd1.out‚6.5 network.recv‚668068! disk.nvme0n1.in‚ 3005.25390625 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 3005.25390625& disk.nvme0c0n1.out‚6106.3349609375" proc.memory.rssMB‚ 2723.0390625( proc.memory.percent‚16.6201114654541 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 52.58203125 disk.sda.in‚0 disk.drbd2165.in‚0Ô¨™ã2§ - _wandbruntime‚2425  _runtime‚2425n~Qƨ À trainer/global_step‚2799 % _timestamp‚1.7843651470381687e+09 _runtime‚ 2425.54410046 _step‚226 & lr-AdamW/pg1‚0.0005607721591877012 & lr-AdamW/pg2‚0.0005607721591877012V9‘Ë© Å % _timestamp‚1.7843651470381687e+09 _runtime‚ 2425.54410046 _step‚226 & lr-AdamW/pg1‚0.0005607721591877012 & lr-AdamW/pg2‚0.0005607721591877012 trainer/global_step‚2799âV\Õ2ª - _wandbruntime‚2425  _runtime‚2425û¤æ%« ÕO BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96798261623296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2978286268928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426834403516416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423888123854848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426834403516416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47975484 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784365147038987e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2946279661568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4907001 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426834045137920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424658559562002 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421717144342528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421717127303168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2945938322432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22097456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96798156531200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5997656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868049 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚426834045137920 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚424658559562002 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421717144342528 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421717127303168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2945938322432 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22097456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96798156531200 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5997656 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22097458 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25868049 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25868049 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚426834045137920 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96798261623296 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2978286268928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚426834403516416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2941432258834 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423888123854848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93819956742144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚426834403516416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2945938322432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47975484 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 $ _timestamp‚1.784365147038987e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25878026 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2946279661568 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4907001 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424035364884480 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4908743 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843651476678379e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96832716435456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2946950949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2979312322560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421863637975040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4908746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424806081287596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2942443312556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2946950949376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2942783862558 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22105088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚228 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5999938 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2979230044160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 2426.17406212 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22105090 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93853385500672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚424035364884480 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4908743 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 % _timestamp‚1.7843651476678379e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2947292295680 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96832716435456 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25887092 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2946950949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5999684 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2979312322560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚421863637975040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4908746 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚426982657180160 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424806081287596 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚424035347845120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10908427 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2942443312556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22105088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2946950949376 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚96832615544832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2942783862558 trainer/global_step‚2799 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22105088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25877101 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚228 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚426982657180160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚424035364884480 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25877101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5999938 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2979230044160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚ 2426.17406212 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22105090 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93853385500672 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424806438876958 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6230644583702087 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18252666294574738 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3703685998916626 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933522343635559 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5348100066184998 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_1_of_1‚0.25383487343788147 particle/accuracy_5_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.49161297082901 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02980944700539112 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011900032870471478 +particle/sensitivity‚0.6108412742614746 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951378107070923 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8553745746612549 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19790104031562805 _step‚230 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5177519917488098 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,jet/accuracy_3_of_3‚0.060958605259656906 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7609978318214417 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.40351730585098267 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166865348816 trainer/global_step‚2799 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.519784152507782 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15201899409294128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.75007563829422 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16563159227371216 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9965688586235046 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6999945640563965 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951324462890625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5575129389762878 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.519784152507782 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12174316495656967 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19258438050746918 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3059525787830353 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6375521421432495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5917708277702332 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11969225853681564 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8390243053436279 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15974989533424377 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004748380742967129 *jet/accuracy_2_of_4‚0.7655235528945923 jet/accuracy_3_of_5‚1 particle/accuracy_2_of_5‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9617108106613159 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 .particle/accuracy_4_of_4‚0.951644778251648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23926055431365967 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5246866941452026 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12174316495656967 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12827584147453308 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.009931244887411594 particle/accuracy_2_of_4‚1 )loss/val_total_loss‚9.612770611455883 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5929301381111145 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5960419774055481 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_2‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906299114227295 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916428923606873 *jet/accuracy_2_of_2‚0.1144750565290451 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5349517464637756 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6940178275108337 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6442326307296753 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7040554285049438 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533332824707031 *particle/specificity‚0.904312789440155 _runtime‚2443.779834297 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13825976848602295 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.573336124420166 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14736755192279816 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24602581560611725 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917799830436707 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9237552881240845 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6821050643920898 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13553957641124725 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17990779876708984 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022956132888793945 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.30346494913101196 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6110318899154663 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4538089334964752 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1426534205675125 *jet/accuracy_1_of_2‚0.7445120215415955 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47901615500450134 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5103837251663208 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7965041399002075 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1608547866344452 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9028207659721375 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19473852217197418 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.52768474817276 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚6 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9952808618545532 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965227246284485 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9952808618545532 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5751453638076782 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12827584147453308 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920766949653625 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5691739320755005 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10371790826320648 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9939245581626892 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5925413370132446 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5080127716064453 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8412970304489136 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5160260200500488 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6309830546379089 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435189843177795 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926505088806152 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12158684432506561 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19417217373847961 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14960628747940063 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5723176598548889 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880955815315247 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09298533946275711 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22798104584217072 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.874950110912323 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5728887915611267 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10976935923099518 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0OååÀ=· º{ 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.52768474817276 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚6 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9952808618545532 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965227246284485 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9952808618545532 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5751453638076782 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12827584147453308 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920766949653625 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5691739320755005 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10371790826320648 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9939245581626892 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965965747833252 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5925413370132446 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5080127716064453 particle/accuracy_2_of_2‚1 (particle/accuracy‚0.8412970304489136 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5160260200500488 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6309830546379089 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6435189843177795 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926505088806152 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12158684432506561 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19417217373847961 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14960628747940063 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5723176598548889 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880955815315247 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09298533946275711 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22798104584217072 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.874950110912323 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5728887915611267 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10976935923099518 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7404841780662537 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1503518670797348 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1960940659046173 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888684153556824 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.021091613918542862 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.669671356678009 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7781935334205627 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5770446062088013 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008855561725795269 7validation_average_jet_accuracy‚0.30603429675102234 % _timestamp‚1.7843651652715826e+09 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6421892046928406 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2162822037935257 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11687981337308884 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5213800072669983 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_3_of_4‚0.07823073118925095 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6655717492103577 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23494036495685577 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7865079045295715 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6760972738265991 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9966357350349426 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13404911756515503 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7051844000816345 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13617675006389618 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15012028813362122 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9944751262664795 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7404841780662537 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10773429274559021 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 *jet/accuracy_2_of_3‚0.4394567906856537 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9150657653808594 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6321507692337036 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7158514857292175 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 .particle/accuracy_3_of_4‚0.997619092464447 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46528762578964233 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13719601929187775 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.6230644583702087 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18252666294574738 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3703685998916626 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9933522343635559 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5348100066184998 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_1_of_1‚0.25383487343788147 particle/accuracy_5_of_5‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7"Purity/0FRt*SRqqt0FBt/event_purity‚0.49161297082901 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02980944700539112 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011900032870471478 +particle/sensitivity‚0.6108412742614746 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951378107070923 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8553745746612549 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47685182094573975 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19790104031562805 _step‚230 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5177519917488098 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,jet/accuracy_3_of_3‚0.060958605259656906 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7609978318214417 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.40351730585098267 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166865348816 trainer/global_step‚2799 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.519784152507782 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15201899409294128 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7"Purity/1FRt0SRqqt2FBt/event_purity‚0.75007563829422 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16563159227371216 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9965688586235046 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6999945640563965 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951324462890625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5575129389762878 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.519784152507782 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12174316495656967 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19258438050746918 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3059525787830353 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6375521421432495 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5917708277702332 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.11969225853681564 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8390243053436279 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15974989533424377 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004748380742967129 *jet/accuracy_2_of_4‚0.7655235528945923 jet/accuracy_3_of_5‚1 particle/accuracy_2_of_5‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9617108106613159 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 .particle/accuracy_4_of_4‚0.951644778251648 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23926055431365967 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5246866941452026 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12174316495656967 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.12827584147453308 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,jet/accuracy_4_of_4‚0.009931244887411594 particle/accuracy_2_of_4‚1 )loss/val_total_loss‚9.612770611455883 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5929301381111145 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5960419774055481 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_2‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906299114227295 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9916428923606873 *jet/accuracy_2_of_2‚0.1144750565290451 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5349517464637756 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6940178275108337 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6442326307296753 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7040554285049438 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533332824707031 *particle/specificity‚0.904312789440155 _runtime‚2443.779834297 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13825976848602295 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.573336124420166 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14736755192279816 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24602581560611725 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917799830436707 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9237552881240845 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6821050643920898 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13553957641124725 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17990779876708984 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022956132888793945 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.30346494913101196 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6110318899154663 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4538089334964752 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1426534205675125 *jet/accuracy_1_of_2‚0.7445120215415955 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.47901615500450134 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5103837251663208 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7965041399002075 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1608547866344452 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9028207659721375 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19473852217197418æÁ)þ›¸ ‚(:“ î€íÒÑ‚¸ï gpu.0.gpu‚37.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032Ú§E gpu.0.temp‚35.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žº»-´¹ ‚(:© î€íÒØ“áñ disk.dm-1.in‚0 disk.sda2.out‚0 network.sent‚ 4.447461e+06 network.recv‚676122 disk.sda3.in‚0 disk.sda.in‚0 disk.sda3.out‚53.375 disk.dm-0.in‚ 3005.25390625" disk.dm-0.out‚6115.43115234375 disk.rbd1.in‚0 disk.rbd1.out‚6.578125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚53.375# memory_percent‚96.8698501586914$ disk./.usageGB‚589.9064064025879% disk.nvme0n1.out‚6115.43115234375# proc.memory.rssMB‚ 2706.90234375) disk./.usagePercent‚39.58911389812064 disk.dm-1.out‚53.375 disk.rbd0.in‚0! disk.nvme0n1.in‚ 3005.25390625* proc.memory.percent‚16.521620750427246 cpu‚25.979265537534616 proc.cpu.threads‚17' disk.nvme0c0n1.out‚6115.43115234375 disk.drbd2165.out‚0% proc.memory.availableMB‚ 512.84375# disk.nvme0c0n1.in‚ 3005.25390625 disk.sda1.in‚0{9 šº ‚(:’ ý€íÒ¹“í gpu.0.gpu‚73.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚242.43' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.486000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýq»Èû ‚(:¸ ý€íÒÙí¯ï% disk.nvme0n1.out‚6122.06982421875( proc.memory.availableMB‚ 512.33984375# proc.memory.rssMB‚ 2706.90234375 cpu‚25.977531121928724 network.recv‚680258$ disk./.usageGB‚589.9059562683105' disk.nvme0c0n1.out‚6122.06982421875 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 3088.1640625 disk.nvme0n1.in‚ 3088.1640625 disk.sda2.out‚0 disk.sda.out‚ 53.62109375 disk.drbd2165.in‚0* proc.memory.percent‚16.521620750427246 proc.cpu.threads‚17 network.sent‚ 4.491736e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 3088.1640625( disk./.usagePercent‚39.5890836892316 disk.sda3.out‚ 53.62109375" disk.dm-0.out‚6122.06982421875 disk.rbd1.out‚6.640625 disk.sda.in‚0 disk.dm-1.out‚ 53.62109375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚96.87292575836182±K8ž¼ ‚(:“ ŒíÒåÿ²ï gpu.0.gpu‚43.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*"¤½ ‚(:™ ŒíÒþÌ·ñ disk.rbd0.in‚0 disk.nvme0c0n1.out‚6124.875 disk.sda1.in‚0 network.recv‚683280 disk.rbd1.in‚0$ disk./.usageGB‚589.9058876037598 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚6124.875 disk.sda.out‚ 54.015625 cpu‚25.975721072649716 proc.cpu.threads‚17 network.sent‚ 4.49872e+06 disk.dm-1.out‚ 54.015625 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 3088.1640625 disk.sda3.out‚ 54.015625 disk.nvme0n1.in‚ 3088.1640625 disk.sda2.out‚0* proc.memory.percent‚16.521620750427246 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 3088.1640625 disk.sda.in‚0# proc.memory.rssMB‚ 2706.90234375 disk.nvme0n1.out‚6124.875 disk.rbd1.out‚6.640625' proc.memory.availableMB‚ 512.0859375) disk./.usagePercent‚39.58907908109598 disk.sda1.out‚0$ memory_percent‚96.87447547912598õÝ |2¾ - _wandbruntime‚2484  _runtime‚2484Ò##yÇ¿ Á _runtime‚2484.129169285 _step‚231 & lr-AdamW/pg1‚0.0005598886382197177 & lr-AdamW/pg2‚0.0005598886382197177 trainer/global_step‚2849 % _timestamp‚1.7843652056231742e+09!a+ÌÀ Æ & lr-AdamW/pg2‚0.0005598886382197177 trainer/global_step‚2849 % _timestamp‚1.7843652056231742e+09 _runtime‚2484.129169285 _step‚231 & lr-AdamW/pg1‚0.0005598886382197177çáhpº2Á - _wandbruntime‚2484  _runtime‚2484’H"#Ü' ÖO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433961450171069 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002524 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11129549 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6133042 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433179724947456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6132781 _step‚232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3014367029760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3014367029760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433961092581819 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3047708382720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091977216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11129813 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26392823 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433179724947456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3014708368896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3010109020861 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599722 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436194450355712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 % _timestamp‚1.7843652056242895e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091977216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433179741986816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22599724 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088Õ á'à ÛO `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95507306768384 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433961450171069 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49002524 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26402800 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26402800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11129549 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6133042 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433179724947456 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6132781 _step‚232 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22599724 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3014367029760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3014367029760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433961092581819 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3047708382720 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚436194091977216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11129813 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26392823 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22599722 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433179724947456 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3014708368896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3010109020861 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22599722 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚436194450355712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 % _timestamp‚1.7843652056242895e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚436194091977216 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚433179741986816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22599724 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0è}ñÛe2Ä - _wandbruntime‚2484  _runtime‚2484ÏßùD_Å ßM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3015721003008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019222 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431097834782720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26411866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26401875 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2484.762778196 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524693 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433326983016448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434108614307413 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433326983016448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433326965977088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.a&€ÿ؆ll.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436342704019456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22607356 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6135148 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401875 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379656704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4998513 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95540754139136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438ddâ"ê&Æ äM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚433326983016448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚433326965977088 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚436342704019456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22607356 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6135148 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26401875 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3015379656704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4998513 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95540754139136 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11133400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3015721003008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019222 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚431097834782720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26411866 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26401875 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2484.762778196 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3010779524693 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚433326983016448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95540735526912 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚434108614307413 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0é †B/2Ç - _wandbruntime‚2484  _runtime‚2484ŒZ:hÈ € trainer/global_step‚2849 0loss/FRt1/assignment_loss‚2.1312224839766003 0loss/FRt2/assignment_loss‚1.0537375214114275 1loss/SRqqt1/assignment_loss‚0.640117434980115 1loss/FBt2/assignment_loss‚0.07292034459146782 /loss/FRt1/detection_loss‚0.5676815767685934 /loss/FBt2/detection_loss‚0.9760949471001995 2loss/SRqqt2/assignment_loss‚0.5642749138677958 /loss/FRt2/detection_loss‚1.4331932402678749 0loss/SRqqt1/detection_loss‚1.031915138017386 1loss/SRqqt2/detection_loss‚0.6721197823013715 /loss/FBt1/detection_loss‚0.5137108012058889 %loss/total_loss‚9.667774306516948 % _timestamp‚1.7843652062582965e+09 _runtime‚2484.763974334 _step‚234 2loss/FBt1/assignment_loss‚0.010786122028227763 epoch‚7Þ¶— É … 0loss/FRt2/assignment_loss‚1.0537375214114275 1loss/SRqqt1/assignment_loss‚0.640117434980115 1loss/FBt2/assignment_loss‚0.07292034459146782 /loss/FRt1/detection_loss‚0.5676815767685934 /loss/FBt2/detection_loss‚0.9760949471001995 2loss/SRqqt2/assignment_loss‚0.5642749138677958 /loss/FRt2/detection_loss‚1.4331932402678749 0loss/SRqqt1/detection_loss‚1.031915138017386 1loss/SRqqt2/detection_loss‚0.6721197823013715 /loss/FBt1/detection_loss‚0.5137108012058889 %loss/total_loss‚9.667774306516948 % _timestamp‚1.7843652062582965e+09 _runtime‚2484.763974334 _step‚234 2loss/FBt1/assignment_loss‚0.010786122028227763 epoch‚7 trainer/global_step‚2849 0loss/FRt1/assignment_loss‚2.1312224839766003êçYR#’Ê ‚(:‡ ›íÒËó‡ð gpu.0.gpu‚32.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚141.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ·Ü‘¼Ë ‚(:± ›íÒ×à®ò disk.sda1.in‚0 network.recv‚686730) disk./.usagePercent‚39.58921630113433$ disk./.usageGB‚589.9079322814941 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda.out‚ 54.2421875$ disk.nvme0n1.out‚6129.3251953125 disk.sda3.out‚ 54.2421875 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚6129.3251953125$ memory_percent‚96.87321186065674 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3088.1640625 disk.dm-1.out‚ 54.2421875 disk.drbd2165.in‚0 disk.rbd1.out‚6.71875( proc.memory.availableMB‚ 512.29296875# proc.memory.rssMB‚ 2706.90234375* proc.memory.percent‚16.521620750427246 cpu‚25.97425541678029 proc.cpu.threads‚17 network.sent‚ 4.501642e+06& disk.nvme0c0n1.out‚6129.3251953125 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3088.1640625 disk.dm-0.in‚ 3088.1640625 disk.sda2.out‚0ËÔ!“Ì ‚(:ˆ ªíÒçíï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚260.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ,¡ ÀÍ ‚(:µ ªíÒÌí¿ñ$ disk./.usageGB‚589.9078826904297 disk.sda2.out‚0 network.sent‚ 4.566438e+06$ memory_percent‚96.87376022338867 disk.sda.out‚ 54.58984375 disk.drbd2165.in‚0* proc.memory.percent‚16.521620750427246" disk.nvme0c0n1.in‚ 3088.1640625 disk.sda3.in‚0 disk.dm-1.out‚ 54.58984375 disk.drbd2165.out‚0" disk.dm-0.out‚6132.66357421875 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 54.58984375 disk.rbd0.out‚0 network.recv‚690849 disk.dm-0.in‚ 3088.1640625# proc.memory.rssMB‚ 2706.90234375 proc.cpu.threads‚17) disk./.usagePercent‚39.58921297303639 disk.nvme0n1.in‚ 3088.1640625 disk.rbd1.out‚6.71875 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.97245011680363% disk.nvme0n1.out‚6132.66357421875 disk.rbd1.in‚0' disk.nvme0c0n1.out‚6132.66357421875& proc.memory.availableMB‚ 512.203125l–Ó“Î ‚(:ˆ ¹íÒ”ÿ–ï gpu.0.gpu‚43.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚150.197' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èZ=»Ï ‚(:° ¹íÒµ•ñ disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 54.97265625( proc.memory.availableMB‚ 512.20703125% disk.nvme0n1.out‚6137.71630859375 disk.sda3.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 3173.15625' disk.nvme0c0n1.out‚6137.71630859375 disk.sda1.in‚0* proc.memory.percent‚16.521620750427246 disk.dm-1.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 4.571405e+06( disk./.usagePercent‚39.5892183491946 disk.rbd0.out‚0$ disk./.usageGB‚589.9079627990723 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚6.78125 disk.nvme0c0n1.in‚ 3173.15625 disk.sda.in‚0 disk.sda3.out‚ 54.97265625$ memory_percent‚96.87373638153076 disk.dm-1.out‚ 54.97265625 disk.dm-0.in‚ 3173.15625# proc.memory.rssMB‚ 2706.90234375 cpu‚25.97094957183209 network.recv‚694207" disk.dm-0.out‚6137.71630859375 disk.sda1.out‚0ùu¤€2Ð - _wandbruntime‚2524  _runtime‚2524QlãÇÑ Á _runtime‚2524.820361134 _step‚235 & lr-AdamW/pg1‚0.0005589878477663329 & lr-AdamW/pg2‚0.0005589878477663329 trainer/global_step‚2899 % _timestamp‚1.7843652463143988e+09 4kDÌÒ Æ _step‚235 & lr-AdamW/pg1‚0.0005589878477663329 & lr-AdamW/pg2‚0.0005589878477663329 trainer/global_step‚2899 % _timestamp‚1.7843652463143988e+09 _runtime‚2524.820361134ë=Ê2Ó - _wandbruntime‚2524  _runtime‚2524–ã^ Ý'Ô ×O _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3099146921984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440541793468416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11321497 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100277990417408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3065403870720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26847275 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3065062531584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11321239 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97178763307008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26847275 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6237221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26857252 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3065062531584 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26857252 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440541776429056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441337242662327 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443607197339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚440541793468416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3065403870720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100277891616768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441337600251577 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22981324 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443607197339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443606838960640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438276874743808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2899 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5084021 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6237476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981324 _runtime‚2524.821812353 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838960640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438276857704448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3060725507769 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5084018 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22981322 % _timestamp‚1.7843652463155637e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828597 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169°Õ ÜO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5084018 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22981322 % _timestamp‚1.7843652463155637e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49828597 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3099146921984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚440541793468416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11321497 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚100277990417408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3065403870720 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49838576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26847275 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3065062531584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11321239 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97178763307008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.pe49Ïak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26847275 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6237221 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97178744694784 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26857252 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3065062531584 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26857252 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚440541776429056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441337242662327 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443607197339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚440541793468416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3065403870720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚100277891616768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441337600251577 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22981324 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚443607197339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443606838960640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚438276874743808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2899 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5084021 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6237476 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22981324 _runtime‚2524.821812353 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443606838960640 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚438276857704448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3060725507769 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3ìÁö2Ö - _wandbruntime‚2525  _runtime‚2525 (´Aç&× áM GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3066416504832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3100179553792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26856327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438423368376320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11325347 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443755451002880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011601 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26866318 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11325094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26866318 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3100259735040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416504832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441485121977283 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5085763 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6239331 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97212192065536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843652469477117e+09 _runtime‚2525.453870141 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49845281 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855274 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0©W;sì&Ø æM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3061736561603 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 trainer/global_step‚2899 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11325094 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26866318 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3100259735040 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3066416504832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441485121977283 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5085763 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6239331 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97212192065536 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843652469477117e+09 _runtime‚2525.453870141 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49845281 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855274 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚438423385415680 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3066416504832 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5085766 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3100179553792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26856327 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚438423368376320 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11325347 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443755451002880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855274 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3061396011601 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26866318 PåÌä Æ & lr-AdamW/pg1‚0.0005580698456836342 & lr-AdamW/pg2‚0.0005580698456836342 trainer/global_step‚2949 % _timestamp‚1.7843652872980978e+09 _runtime‚2565.804734583 _step‚239ïÌDKI2å - _wandbruntime‚2565  _runtime‚2565.Q<ƒ‘æ ÖO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3116099372544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚2è'uK&69 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3111341994677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98850201233408 _step‚240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451019585944064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102000713955328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332085 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585944064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3115758033408 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27301727 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451019944322560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11512648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102000610960384 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3150512721920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚2565.805028748 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98850182621184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362922 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362922 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451019944322560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5171271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445602391298048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6341377 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6341108 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5171268õµá'ç ÛO UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447903844950016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 # _timestamp‚1.78436528729882e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3115758033408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451019585944064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3115758033408 trainer/global_step‚2949 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27301727 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451019944322560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11512648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23362924 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102000610960384 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3150512721920 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚2565.805028748 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚98850182621184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚447903844950016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23362922 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23362922 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451019944322560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5171271 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445602391298048 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6341377 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6341108 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5171268 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3116099372544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3111341994677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98850201233408 _step‚240 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚451019585944064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102000713955328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚448713750332085 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447903827910656 I¬§2è - _wandbruntime‚2566  _runtime‚2566•_(ç&é áM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3116770660352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448861272057791 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98883629991936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3151534274048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27320770 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691326 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23370556 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11516211 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6343446 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445748901969920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448051068940288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112006656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11516462 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6343198 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3116770660352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691326 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112353048511 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚241 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3117112006656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102035054986752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681333 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27310779 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169G$¢8˜ ê æM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784365287929128e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5173016 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448051068940288 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3117112006656 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11516462 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6343198 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3116770660352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚50691326 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23370554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112353048511 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448051068940288 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚241 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3117112006656 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102035054986752 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445748919009280 C6$F½TDeviceStatsMonitor.on_train_batch_end/active.all.freed‚50681333 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27310779 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3116770660352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚448861272057791 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5173013 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98883629991936 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3151534274048 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27320770 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691326 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23370556 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11516211 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6343446 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445748901969920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EK2ë - _wandbruntime‚2566  _runtime‚2566>70ôì ú .loss/FRt1/assignment_loss‚2.18622197008278 1loss/FBt2/assignment_loss‚0.04537979719645913 /loss/FRt1/detection_loss‚0.5718510880701919 epoch‚7 % _timestamp‚1.7843652879309354e+09 /loss/FRt2/assignment_loss‚1.136575581776749 2loss/SRqqt2/assignment_loss‚0.5639859929217546 /loss/FRt2/detection_loss‚1.5264313439109365 1loss/SRqqt1/detection_loss‚0.9238178709241015 trainer/global_step‚2949 _runtime‚2566.436627985 _step‚242 1loss/FBt1/assignment_loss‚0.04419320973433686 1loss/SRqqt2/detection_loss‚0.6927333015689339 /loss/FBt1/detection_loss‚0.3733481537025895 .loss/FBt2/detection_loss‚1.168510538547538 0loss/SRqqt1/assignment_loss‚0.56146228310322 $loss/total_loss‚9.79451113153959¸Û9ãí ÿ /loss/FRt1/detection_loss‚0.5718510880701919 epoch‚7 % _timestamp‚1.7843652879309354e+09 /loss/FRt2/assignment_loss‚1.136575581776749 2loss/SRqqt2/assignment_loss‚0.5639859929217546 /loss/FRt2/detection_loss‚1.5264313439109365 1loss/SRqqt1/detection_loss‚0.9238178709241015 trainer/global_step‚2949 _runtime‚2566.436627985 _step‚242 1loss/FBt1/assignment_loss‚0.04419320973433686 1loss/SRqqt2/detection_loss‚0.6927333015689339 /loss/FBt1/detection_loss‚0.3733481537025895 .loss/FBt2/detection_loss‚1.168510538547538 0loss/SRqqt1/assignment_loss‚0.56146228310322 $loss/total_loss‚9.79451113153959 .loss/FRt1/assignment_loss‚2.18622197008278 1loss/FBt2/assignment_loss‚0.04537979719645913òxYÛ·’î ‚(:‡ õíÒÍæÂî gpu.0.gpu‚79.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚231.105' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.221 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­bœ£½ï ‚(:² õíÒÅÉ«ï disk.sda2.out‚0' disk.nvme0c0n1.out‚6159.87939453125 network.sent‚ 4.647265e+06 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0( proc.memory.availableMB‚ 512.27734375) disk./.usagePercent‚39.58917175582337 disk.rbd0.in‚0# proc.memory.rssMB‚ 2706.90234375 cpu‚25.96455446546762 disk.rbd0.out‚0 network.recv‚708929$ disk./.usageGB‚589.9072685241699 disk.sda.out‚ 56.140625* proc.memory.percent‚16.521620750427246 disk.sda3.out‚ 56.140625 disk.drbd2165.in‚0% disk.nvme0n1.out‚6159.87939453125" disk.nvme0c0n1.in‚ 3209.3671875$ memory_percent‚96.87330722808838 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 56.140625 disk.rbd1.out‚6.921875 disk.sda.in‚0 disk.nvme0n1.in‚ 3209.3671875 disk.dm-0.in‚ 3209.3671875" disk.dm-0.out‚6159.87939453125 disk.sda2.in‚0qÙžð ‚(:“ „‚íÒáÿøì gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rµšjÀñ ‚(:µ „‚íÒ§óð" disk.dm-0.out‚6162.93310546875$ memory_percent‚96.87483310699463) disk./.usagePercent‚39.58916996377063" disk.nvme0c0n1.in‚ 3251.4140625 disk.dm-1.out‚ 56.5234375 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3251.4140625% disk.nvme0n1.out‚6162.93310546875( proc.memory.availableMB‚ 512.02734375# proc.memory.rssMB‚ 2706.90234375* proc.memory.percent‚16.521620750427246' disk.nvme0c0n1.out‚6162.93310546875 disk.sda3.out‚ 56.5234375 cpu‚25.96292820298601 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 56.5234375 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 4.710176e+06 network.recv‚713292$ disk./.usageGB‚589.9072418212891 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚6.921875 disk.drbd2165.in‚0 disk.dm-0.in‚ 3251.4140625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0côgf2ò - _wandbruntime‚2606  _runtime‚2606¥„£}Çó Á & lr-AdamW/pg1‚0.0005571346909331803 & lr-AdamW/pg2‚0.0005571346909331803 trainer/global_step‚2999 % _timestamp‚1.7843653281085548e+09 _runtime‚2606.614532385 _step‚243°®*"Ìô Æ & lr-AdamW/pg2‚0.0005571346909331803 trainer/global_step‚2999 % _timestamp‚1.7843653281085548e+09 _runtime‚2606.614532385 _step‚243 & lr-AdamW/pg1‚0.0005571346909331803ó˜5 |2õ - _wandbruntime‚2606  _runtime‚2606…iÝ#ß'ö ÙO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458432332927488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456089900412593 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27766156 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455265896431616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794874368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458432691305984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100521620547584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452927941931008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3166453535232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332927488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452927924891648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11703672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3201581639680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27756179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100521639159808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481585 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784365328109751e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3161617931695 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23744522 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _runtime‚2606.615900367 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3166794874368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27766156 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6445151 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6444881 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455265879392256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456089542823343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455265896431616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455265879392256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458432691305984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚2999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500701 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚244 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23744524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23744524 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103723202187264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0P*¹[ä'÷ ÞO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23744522 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3166453535232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11703399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27756179 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5258521 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201666022400 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚103723305182208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚455265896431616 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3166794874368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚458432691305984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100521620547584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452927941931008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3166453535232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚458432332927488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452927924891648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11703672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3201581639680 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27756179 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100521639159808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3161958481585 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784365328109751e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3161617931695 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23744522 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _runtime‚2606.615900367 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3166794874368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27766156 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6445151 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6444881 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚455265879392256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456089542823343 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455265896431616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚455265879392256 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚458432691305984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚2999 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51500701 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚244 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23744524 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5258518 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23744524 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚103723202187264 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚458432332927488 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456089900412593 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27766156 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944969728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455413120421888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103757657233920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100555067918336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784365328737065e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458580944969728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11707194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527378 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3167466162176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466162176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517385 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11707454 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517385 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527378 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5260266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455413137461248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2607.243225394 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453074452602880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6446931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6447188 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138299 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3162628985417 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3202692303360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456237064548937 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3202607927808ŠìîÆî&ú èM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455413137461248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3202607927808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458580944969728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455413120421888 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚103757657233920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100555067918336 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784365328737065e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458580944969728 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11707194 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455413120421888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527378 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3167466162176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3167466162176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51517385 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11707454 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517385 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527378 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5260266 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455413137461248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2607.243225394 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453074452602880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27775222 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6446931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6447188 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456237422138299 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3162628985417 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23752154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3202692303360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23752156 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456237064548937õ¾½¸Z2û - _wandbruntime‚2607  _runtime‚2607ˆ>œü € 1loss/SRqqt2/detection_loss‚0.6888294273409091 /loss/FBt1/detection_loss‚0.4825386577285826 /loss/FBt2/detection_loss‚0.7324827083283001 epoch‚7 _step‚246 /loss/FRt1/detection_loss‚0.5604160850853436 %loss/total_loss‚9.855950309859425 /loss/FRt1/assignment_loss‚2.229484227955753 2loss/SRqqt1/assignment_loss‚0.7037661403113048 2loss/SRqqt2/assignment_loss‚0.6524387325584793 2loss/FBt1/assignment_loss‚0.008703261723214097 0loss/SRqqt1/detection_loss‚1.132016706723054 % _timestamp‚1.7843653287387273e+09 _runtime‚2607.244409128 0loss/FRt2/assignment_loss‚1.1470648055424506 2loss/FBt2/assignment_loss‚0.009047004935542842 .loss/FRt2/detection_loss‚1.509162551626491 trainer/global_step‚2999ðºàÑ ý … 2loss/SRqqt2/assignment_loss‚0.6524387325584793 2loss/FBt1/assignment_loss‚0.008703261723214097 0loss/SRqqt1/detection_loss‚1.132016706723054 % _timestamp‚1.7843653287387273e+09 _runtime‚2607.244409128 0loss/FRt2/assignment_loss‚1.1470648055424506 2loss/FBt2/assignment_loss‚0.009047004935542842 .loss/FRt2/detection_loss‚1.509162551626491 trainer/global_step‚2999 1loss/SRqqt2/detection_loss‚0.6888294273409091 /loss/FBt1/detection_loss‚0.4825386577285826 /loss/FBt2/detection_loss‚0.7324827083283001 epoch‚7 _step‚246 /loss/FRt1/detection_loss‚0.5604160850853436 %loss/total_loss‚9.855950309859425 /loss/FRt1/assignment_loss‚2.229484227955753 2loss/SRqqt1/assignment_loss‚0.7037661403113048öyÔŸ’þ ‚(:‡ “‚íÒ„Óï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ©µ¹ÿ ‚(:® “‚íÒÝÿäñ& disk.nvme0c0n1.out‚6168.0341796875 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 56.82421875 network.recv‚716896( disk./.usagePercent‚39.5892168131494 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚0$ disk.nvme0n1.out‚6168.0341796875 network.sent‚ 4.714993e+06" disk.nvme0c0n1.in‚ 3251.4140625 disk.dm-0.in‚ 3251.4140625 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.961491815111575 disk.rbd1.out‚7! disk.dm-0.out‚6168.0341796875$ memory_percent‚96.87211513519287* proc.memory.percent‚16.521620750427246 disk.sda.out‚ 56.82421875 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 3251.4140625# proc.memory.rssMB‚ 2706.90234375 proc.cpu.threads‚17$ disk./.usageGB‚589.9079399108887 disk.sda3.out‚ 56.82421875 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 512.47265625¼Œ¹€“€ ‚(:ˆ ¢‚íÒŽ°í gpu.0.gpu‚49.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öaè¹ ‚(:® ¢‚íÒ…Ø÷ï network.sent‚ 4.777875e+06 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 57.1796875 network.recv‚721201 disk.sda2.out‚0 disk.rbd0.out‚0 cpu‚25.959954957534602 disk.rbd0.in‚0) disk./.usagePercent‚39.58919991665214 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 3251.4140625% disk.nvme0n1.out‚6171.09130859375$ memory_percent‚96.87514305114746 disk.sda.out‚ 57.1796875 disk.rbd1.in‚0 disk.rbd1.out‚7 disk.dm-1.out‚ 57.1796875' proc.memory.availableMB‚ 511.9765625 disk.dm-0.in‚ 3251.4140625$ disk./.usageGB‚589.9076881408691 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2706.90234375" disk.nvme0c0n1.in‚ 3251.4140625' disk.nvme0c0n1.out‚6171.09130859375" disk.dm-0.out‚6171.09130859375* proc.memory.percent‚16.521620750427246è¤%“‚ ‚(:ˆ ±‚íÒŽÏ·í gpu.0.gpu‚86.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚257.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LT™¹ƒ ‚(:® ±‚íÒÈ·Åï* proc.memory.percent‚16.521620750427246 disk.dm-1.out‚57.53125% disk.nvme0n1.out‚6175.97216796875 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 4.782905e+06* disk./.usagePercent‚39.589060392545974 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3251.4140625 disk.dm-0.in‚ 3251.4140625 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 3251.4140625 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda.in‚0$ memory_percent‚96.87364101409912# proc.memory.rssMB‚ 2706.90234375 cpu‚25.95840581866979" disk.dm-0.out‚6175.97216796875 disk.sda.out‚57.53125 network.recv‚724958$ disk./.usageGB‚589.9056091308594 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 512.22265625 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚7.0625 disk.sda3.out‚57.53125 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6175.97216796875 proc.cpu.threads‚17É%Ì×2„ - _wandbruntime‚2647  _runtime‚2647Ÿø Å… ¿ % _timestamp‚1.7843653690937426e+09 _runtime‚2647.599716561 _step‚247 % lr-AdamW/pg1‚0.000556182443578213 % lr-AdamW/pg2‚0.000556182443578213 trainer/global_step‚3049Ä °Öʆ Ä % lr-AdamW/pg2‚0.000556182443578213 trainer/global_step‚3049 % _timestamp‚1.7843653690937426e+09 _runtime‚2647.599716561 _step‚247 % lr-AdamW/pg1‚0.000556182443578213÷½RU2‡ - _wandbruntime‚2647  _runtime‚2647&œ¢‚ ˆ ÚO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079910912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6548454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚3049 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252766000128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24126124 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24126122 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462627930873856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3212574968493 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465845438289408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28220608 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3217490376192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3252850382848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465845438289408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52346732 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105445824474112 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460253458485248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462627947913216 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3217490376192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24126122 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269ky+å'‰ ßO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5345771 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105445927469056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102193058473984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11894225 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚463466050493101 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2647.600956041 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24126124 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3217149037056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3217490376192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24126122 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465845079910912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6548454 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚3049 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3252766000128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24126124 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24126122 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚462627930873856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3212574968493 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465845438289408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28220608 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3217490376192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3252850382848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465845438289408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52346732 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105445824474112 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚460253458485248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚462627947913216 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24133756 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _runtime‚2648.228683198 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465993691953152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462775188942848 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11898067 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460399986196480 trainer/global_step‚3049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465993691953152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462775171903488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105480376735232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253870890496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3218161664000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28229674 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6550551 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6550294 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843653697224588e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11897807 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3213586022327 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363430 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28219683 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5347516 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28219683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353437 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333567488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24133754 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3218161664000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304]E­Å#Œ éM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3218161664000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28229674 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6550551 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6550294 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843653697224588e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102226505844736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11897807 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3213586022327 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363430 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28219683 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5347516 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28219683 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52353437 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465993333567488 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24133754 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3218161664000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52353437 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105480267456000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5347513 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3218503010304 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363430 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24133756 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 _runtime‚2648.228683198 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465993691953152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462775188942848 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11898067 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚460399986196480 trainer/global_step‚3049 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24133756 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚46599369ÒÄJàç1953152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462775171903488 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚105480376735232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3253870890496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24133754 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ùt S2 - _wandbruntime‚2648  _runtime‚2648*BÑ}Ž  epoch‚7 0loss/FRt1/assignment_loss‚2.1640244460687406 2loss/SRqqt2/assignment_loss‚0.5681143870814851 1loss/FBt1/assignment_loss‚0.08204864707593494 /loss/FRt1/detection_loss‚0.5947297268005503 1loss/SRqqt1/detection_loss‚1.0290800690553272 1loss/SRqqt2/detection_loss‚0.6662875522906023 /loss/FBt1/detection_loss‚0.4243789787298806 trainer/global_step‚3049 % _timestamp‚1.7843653697242355e+09 _runtime‚2648.229932917 _step‚250 /loss/FBt2/detection_loss‚1.6749254191915195 &loss/total_loss‚10.323786189419838 2loss/SRqqt1/assignment_loss‚0.5902294778040077 /loss/FRt2/assignment_loss‚1.034290402150545 1loss/FBt2/assignment_loss‚0.01763318564913637 /loss/FRt2/detection_loss‚1.4780438975221095ßͬ®  † /loss/FBt2/detection_loss‚1.6749254191915195 &loss/total_loss‚10.323786189419838 2loss/SRqqt1/assignment_loss‚0.5902294778040077 /loss/FRt2/assignment_loss‚1.034290402150545 1loss/FBt2/assignment_loss‚0.01763318564913637 /loss/FRt2/detection_loss‚1.4780438975221095 epoch‚7 0loss/FRt1/assignment_loss‚2.1640244460687406 2loss/SRqqt2/assignment_loss‚0.5681143870814851 1loss/FBt1/assignment_loss‚0.08204864707593494 /loss/FRt1/detection_loss‚0.5947297268005503 1loss/SRqqt1/detection_loss‚1.0290800690553272 1loss/SRqqt2/detection_loss‚0.6662875522906023 /loss/FBt1/detection_loss‚0.4243789787298806 trainer/global_step‚3049 % _timestamp‚1.7843653697242355e+09 _runtime‚2648.229932917 _step‚250ú,/.$“ ‚(:ˆ À‚íÒ÷ð gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚112.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5416 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#6³‘ ‚(:¨ À‚íÒŠ´¯ó disk.sda3.out‚ 57.6796875 disk.dm-1.out‚ 57.6796875 disk.rbd0.in‚0 disk.dm-0.in‚ 3326.515625 disk.dm-0.out‚ 6179.34765625 disk.sda2.out‚0$ disk.nvme0c0n1.out‚ 6179.34765625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 3326.515625( proc.memory.availableMB‚ 512.64453125 disk.sda1.out‚0" disk.nvme0n1.out‚ 6179.34765625$ memory_percent‚96.87106609344482* proc.memory.percent‚16.521620750427246 disk.sda3.in‚0 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 3326.515625) disk./.usagePercent‚39.58920375676515 disk.sda.in‚0 disk.drbd2165.out‚0 cpu‚25.956975280863922 network.sent‚ 4.800903e+06 network.recv‚728628 disk.dm-1.in‚0 disk.rbd1.out‚7.0625$ disk./.usageGB‚589.9077453613281 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17# proc.memory.rssMB‚ 2706.90234375 disk.sda.out‚ 57.6796875@56>ž’ ‚(:“ Ï‚íÒËôÃî gpu.0.gpu‚44.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý˜[»“ ‚(:° Ï‚íÒšŽð( disk./.usagePercent‚39.5891883963131 disk.drbd2165.out‚0 disk.rbd0.in‚0 network.sent‚ 4.850683e+06$ disk./.usageGB‚589.9075164794922 disk.dm-1.in‚0 disk.sda3.out‚ 57.95703125 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 3326.515625& disk.nvme0c0n1.out‚6185.2919921875$ memory_percent‚96.87116146087646 network.recv‚733237 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 3326.515625 disk.rbd1.in‚0* proc.memory.percent‚16.521620750427246 disk.sda2.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2706.90234375 cpu‚25.955517575291005 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 3326.515625 disk.sda.out‚ 57.95703125( proc.memory.availableMB‚ 512.62890625 disk.dm-1.out‚ 57.95703125! disk.dm-0.out‚6185.2919921875 disk.rbd1.out‚7.1875 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚6185.2919921875&ØW “” ‚(:ˆ Þ‚íÒŠ…Åî gpu.0.gpu‚86.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q®¢x¸• ‚(:­ Þ‚íÒ¼©Øï disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 58.33984375 cpu‚25.954019404251873$ disk./.usageGB‚589.9077453613281 disk.dm-1.out‚ 58.33984375 disk.nvme0n1.in‚ 3326.515625# disk.nvme0n1.out‚6188.412109375 disk.dm-0.in‚ 3326.515625 disk.sda2.out‚0 disk.rbd1.in‚0 network.recv‚736171 disk.drbd2165.out‚0* proc.memory.percent‚16.521620750427246 disk.sda3.in‚0 network.sent‚ 4.855597e+06 disk.dm-0.out‚6188.412109375% disk.nvme0c0n1.out‚6188.412109375 disk.sda1.in‚0) disk./.usagePercent‚39.58920375676515 disk.sda3.out‚ 58.33984375 disk.dm-1.in‚0 disk.rbd1.out‚7.1875! disk.nvme0c0n1.in‚ 3326.515625# memory_percent‚96.8703031539917 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 512.76953125 proc.cpu.threads‚17# proc.memory.rssMB‚ 2706.90234375¢Ž,ý2– - _wandbruntime‚2688  _runtime‚2688ãÕ Ç— Á _step‚251 & lr-AdamW/pg1‚0.0005552131647798005 & lr-AdamW/pg2‚0.0005552131647798005 trainer/global_step‚3099 % _timestamp‚1.7843654101162376e+09 _runtime‚2688.622200989’¹”̘ Æ & lr-AdamW/pg1‚0.0005552131647798005 & lr-AdamW/pg2‚0.0005552131647798005 trainer/global_step‚3099 % _timestamp‚1.7843654101162376e+09 _runtime‚2688.622200989 _step‚251ûG"¡F2™ - _wandbruntime‚2688  _runtime‚2688@®öß'š ÙO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5433021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5433018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826894336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473258185272832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚3099 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28665083 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984359 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2688.626746661 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3268185878016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12084900 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3267844538880 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6652139 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168591364608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6651882 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473258185272832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3267844538880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3303998260736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3304076352000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263191455401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103864496400384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467578992078848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3268185878016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53172805 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473257826894336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28665083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3262850905511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843654101173022e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469989999394816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24507722 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚252 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28675060 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469989999394816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467579009118208#Ânä'› ÞO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚470841842984359 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚2688.626746661 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3268185878016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12084900 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24507724 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3267844538880 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6652139 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107168591364608 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚78091264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6651882 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473258185272832 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3267844538880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3303998260736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3304076352000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3263191455401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚103864496400384 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚467578992078848 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3268185878016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53172805 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473257826894336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28665083 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3262850905511 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843654101173022e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚469989999394816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24507722 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚252 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28675060 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚469989999394816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚467579009118208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5433021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5433018 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473257826894336 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473258185272832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚3099 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28665083 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12088733 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959233 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470137240424448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6653970 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3305107337216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467725519790080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674135 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24515354 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470989722299315 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28684126 H?DeviceStatsMonit,/vior.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24515356 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198512128 % _timestamp‚1.7843654107477891e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198512128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473406438936576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚253 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107203051108352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470137223385088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5434763 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2689.253994422 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189489gÖØî&ž èM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚470137223385088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5434763 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2689.253994422 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚470137223385088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚470137240424448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53189489 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24515356 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5434766 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12088733 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3263861959233 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚470137240424448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6653970 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3305107337216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚467725519790080 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28674135 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24515354 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚470989722299315 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103897925158912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28684126 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103897943771136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24515356 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3269198512128 % _timestamp‚1.7843654107477891e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3269198512128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473406438936576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚75986944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚253 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53199482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107203051108352 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ýÇóÁ2Ÿ - _wandbruntime‚2689  _runtime‚2689iç  € 0loss/FRt2/assignment_loss‚1.0304242531061172 1loss/FBt2/assignment_loss‚0.07953362959944808 %loss/total_loss‚9.163580543652753 % _timestamp‚1.7843654107498865e+09 0loss/FRt1/assignment_loss‚2.0404335228824912 2loss/FBt1/assignment_loss‚0.030544347034127475 .loss/FRt2/detection_loss‚1.382865825522691 1loss/SRqqt2/detection_loss‚0.6620888978695502 /loss/FBt2/detection_loss‚0.7547101020078424 epoch‚7 trainer/global_step‚3099 _runtime‚2689.255573964 2loss/SRqqt1/assignment_loss‚0.6063345030289873 2loss/SRqqt2/assignment_loss‚0.5567754086427269 .loss/FRt1/detection_loss‚0.603342601008746 /loss/FBt1/detection_loss‚0.4265754059312605 _step‚254 1loss/SRqqt1/detection_loss‚0.9899520470187642zƒà˜ ¡ … 1loss/SRqqt1/detection_loss‚0.9899520470187642 0loss/FRt2/assignment_loss‚1.0304242531061172 1loss/FBt2/assignment_loss‚0.07953362959944808 %loss/total_loss‚9.163580543652753 % _timestamp‚1.7843654107498865e+09 0loss/FRt1/assignment_loss‚2.0404335228824912 2loss/FBt1/assignment_loss‚0.030544347034127475 .loss/FRt2/detection_loss‚1.382865825522691 1loss/SRqqt2/detection_loss‚0.6620888978695502 /loss/FBt2/detection_loss‚0.7547101020078424 epoch‚7 trainer/global_step‚3099 _runtime‚2689.255573964 2loss/SRqqt1/assignment_loss‚0.6063345030289873 2loss/SRqqt2/assignment_loss‚0.5567754086427269 .loss/FRt1/detection_loss‚0.603342601008746 /loss/FBt1/detection_loss‚0.4265754059312605 _step‚254þUÙ²"ž¢ ‚(:“ í‚íÒ¼Úþï gpu.0.gpu‚69.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cÿ­!¹£ ‚(:® í‚íÒé•ò network.sent‚ 4.872699e+06 disk.sda3.out‚ 58.7109375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3326.515625 disk.dm-0.in‚ 3326.515625" disk.dm-0.out‚6193.94580078125! disk.nvme0c0n1.in‚ 3326.515625# proc.memory.rssMB‚ 2706.90234375$ disk./.usageGB‚589.9068222045898% disk.nvme0n1.out‚6193.94580078125 disk.rbd1.out‚7.3125 disk.sda.out‚ 58.7109375 cpu‚25.95247868747006 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6193.94580078125& proc.memory.availableMB‚ 512.984375) disk./.usagePercent‚39.58914180294187 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.521620750427246 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0 network.recv‚738768 disk.rbd1.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 58.7109375 disk.sda3.in‚0$ memory_percent‚96.86899185180664 proc.cpu.threads‚17 5ñp’¤ ‚(:‡ ü‚íÒ“¤•ï gpu.0.gpu‚26.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†.¥Ö¿¥ ‚(:´ ü‚íÒåºÉñ$ disk./.usageGB‚589.9067649841309$ disk.nvme0n1.out‚6196.9697265625 cpu‚25.951200683636365 disk.sda.out‚ 58.90234375 disk.sda2.in‚0$ memory_percent‚96.86901569366455* proc.memory.percent‚16.521620750427246 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 58.90234375 disk.dm-1.out‚ 58.90234375 disk.dm-0.in‚ 3387.9765625 disk.rbd1.in‚0) disk./.usagePercent‚39.58913796282885 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 3387.9765625 network.recv‚743039 disk.rbd1.out‚7.3125( proc.memory.availableMB‚ 512.98046875 network.sent‚ 4.924487e+06 disk.drbd2165.in‚0! disk.dm-0.out‚6196.9697265625 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 3387.9765625& disk.nvme0c0n1.out‚6196.9697265625 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2706.90234375 ΰ2¦ - _wandbruntime‚2729  _runtime‚2729àÚÌûŧ ¿ & lr-AdamW/pg2‚0.0005542269167929091 trainer/global_step‚3149 $ _timestamp‚1.784365450903156e+09 _runtime‚ 2729.41084357 _step‚255 & lr-AdamW/pg1‚0.0005542269167929091nÙ<¿Ê¨ Ä $ _timestamp‚1.784365450903156e+09 _runtime‚ 2729.41084357 _step‚255 & lr-AdamW/pg1‚0.0005542269167929091 & lr-AdamW/pg2‚0.0005542269167929091 trainer/global_step‚3149ÿ'¥C 2© - _wandbruntime‚2729  _runtime‚2729Íñš>à'ª ÚO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3318881379840 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843654509039512e+09 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚474904542711808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5520271 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3318540040704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108891342808576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105535952939008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5520268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478217993064867 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29129512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6754962 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3149 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29119535 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3355389869568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24889324 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24889322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12275230 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105535934326784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474904525672448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚477352050876416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚477352033837056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _step‚256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018836 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚480670932256256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108891233522176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚478218350654117 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3318540040704 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMo%¸‡¤cnitor.on_train_batch_start/segment.large_pool.allocated‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108891342808576 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105535952939008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5520268 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚478217993064867 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29129512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6754962 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3149 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29119535 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3355389869568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24889324 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24889322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12275230 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105535934326784 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚480670573877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚474904525672448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚477352050876416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚477352033837056 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _step‚256€ó!;“¬ ‚(:ˆ ‹ƒíÒÕÿî gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&]wÁ­ ‚(:¶ ‹ƒíÒìöþð disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚6202.17431640625 disk.rbd1.out‚7.375 network.sent‚ 4.927409e+06" disk.dm-0.out‚6202.17431640625 cpu‚25.949753760576595 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3387.9765625 disk.dm-0.in‚ 3387.9765625" disk.nvme0c0n1.in‚ 3387.9765625$ memory_percent‚96.87047004699707 network.recv‚746617 disk.sda1.out‚0 disk.sda3.out‚ 59.35546875 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚0 disk.sda.out‚ 59.35546875 disk.dm-1.out‚ 59.35546875 disk.rbd0.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 512.7421875' disk.nvme0c0n1.out‚6202.17431640625* disk./.usagePercent‚39.589140522904195$ disk./.usageGB‚589.9068031311035 disk.drbd2165.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2706.90234375* proc.memory.percent‚16.521620750427246Sƒˆ2® - _wandbruntime‚2730  _runtime‚2730Hu}Òê&¯ äM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29128587 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚3149 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24896956 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480818827534336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29138578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475051053383680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025541 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3319552667648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12279016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12279288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3319552667648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3319894013952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480819185920000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477499274866688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035534 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24896954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24896954 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478365514790461 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6757003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105569363085312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3356410954752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3356326579200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475051036344320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108925792652288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚477499274866688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3319894013952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379823 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105569381697536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚2730.043261759 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5522016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6757272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925689664512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269FðãŒï&° éM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54025541 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3319552667648 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035534 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12279016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12279288 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3314818996143 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3319552667648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3319894013952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚480819185920000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚477499274866688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035534 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24896954 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24896954 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚478365514790461 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6757003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105569363085312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3356410954752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3356326579200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚475051036344320 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108925792652288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚477499274866688 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3319894013952 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚478365872379823 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105569381697536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚2730.043261759 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5522016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6757272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108925689664512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29128587 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 trainer/global_step‚3149 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24896956 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚480818827534336 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _step‚257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29138578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚475051053383680 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438Ëóº52± - _wandbruntime‚2730  _runtime‚2730P­M̲ ÿ 2loss/SRqqt2/assignment_loss‚0.6205538152167505 /loss/FRt1/detection_loss‚0.6016895247090127 .loss/FBt2/detection_loss‚1.010269074360966 trainer/global_step‚3149 _step‚258 0loss/FRt2/assignment_loss‚1.0239818362479514 2loss/FBt1/assignment_loss‚0.008616896965891303 0loss/SRqqt1/detection_loss‚1.068793755127365 .loss/FBt1/detection_loss‚0.465173770673573 _runtime‚2730.044403274 epoch‚7 0loss/FRt1/assignment_loss‚2.1128632740101136 2loss/SRqqt1/assignment_loss‚0.6711249561894493 /loss/FRt2/detection_loss‚1.5000349536478679 1loss/SRqqt2/detection_loss‚0.6387250999445279 $loss/total_loss‚9.75190811747037 % _timestamp‚1.7843654515387256e+09 2loss/FBt2/assignment_loss‚0.030081160376900853ÍިΠ³ „ 0loss/FRt2/assignment_loss‚1.0239818362479514 2loss/FBt1/assignment_loss‚0.008616896965891303 0loss/SRqqt1/detection_loss‚1.068793755127365 .loss/FBt1/detection_loss‚0.465173770673573 _runtime‚2730.044403274 epoch‚7 0loss/FRt1/assignment_loss‚2.1128632740101136 2loss/SRqqt1/assignment_loss‚0.6711249561894493 /loss/FRt2/detection_loss‚1.5000349536478679 1loss/SRqqt2/detection_loss‚0.6387250999445279 $loss/total_loss‚9.75190811747037 % _timestamp‚1.7843654515387256e+09 2loss/FBt2/assignment_loss‚0.030081160376900853 2loss/SRqqt2/assignment_loss‚0.6205538152167505 /loss/FRt1/detection_loss‚0.6016895247090127 .loss/FBt2/detection_loss‚1.010269074360966 trainer/global_step‚3149 _step‚258‚¤!Bdž´ ‚(:“ šƒíÒˆ×èï gpu.0.gpu‚76.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÆR¬ºµ ‚(:¯ šƒíÒ罆ò disk.sda2.out‚0$ disk./.usageGB‚589.9077110290527 disk.sda3.out‚ 59.796875 disk.dm-0.in‚ 3387.9765625 disk.sda.in‚0 disk.sda.out‚ 59.796875& proc.memory.availableMB‚ 512.734375 network.recv‚750749 disk.rbd0.in‚0 disk.nvme0n1.in‚ 3387.9765625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3387.9765625 disk.sda1.out‚0 disk.dm-1.out‚ 59.796875$ memory_percent‚96.87051773071289 disk.sda1.in‚0# proc.memory.rssMB‚ 2706.90234375* proc.memory.percent‚16.521620750427246 cpu‚25.948289962224408 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚7.375 disk.sda3.in‚0 disk.dm-1.in‚0* disk./.usagePercent‚39.589201452697345 disk.drbd2165.in‚0" disk.dm-0.out‚6206.15966796875' disk.nvme0c0n1.out‚6206.15966796875 network.sent‚ 4.992414e+06 disk.rbd0.out‚0% disk.nvme0n1.out‚6206.15966796875iÊB”ž¶ ‚(:“ ©ƒíÒÿž¶î gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚253.453' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.690599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Ž‡Ù¬· ‚(:¨ ©ƒíÒú„±ï disk.sda1.in‚0# proc.memory.availableMB‚512.625* proc.memory.percent‚16.521620750427246 disk.nvme0n1.in‚ 3387.9765625* disk./.usagePercent‚39.589089065389814 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 3387.9765625 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.dm-0.in‚ 3387.9765625 disk.rbd1.out‚7.4375 network.sent‚ 5.03222e+06 network.*£öÚrecv‚755164 disk.drbd2165.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚6212.19775390625# proc.memory.rssMB‚ 2706.90234375% disk.nvme0n1.out‚6212.19775390625 disk.rbd1.in‚0' disk.nvme0c0n1.out‚6212.19775390625 disk.sda1.out‚0$ memory_percent‚96.87118530273438$ disk./.usageGB‚589.9060363769531 disk.sda2.out‚0 disk.dm-1.out‚60.15625 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.out‚60.15625 cpu‚25.94700457092422 disk.sda3.out‚60.15625 disk.sda.in‚0ypm2¸ - _wandbruntime‚2770  _runtime‚2770ôïGǹ Á & lr-AdamW/pg1‚0.0005532237629624037 & lr-AdamW/pg2‚0.0005532237629624037 trainer/global_step‚3199 % _timestamp‚1.7843654918982906e+09 _runtime‚2770.404343499 _step‚259¹-ó̺ Æ & lr-AdamW/pg1‚0.0005532237629624037 & lr-AdamW/pg2‚0.0005532237629624037 trainer/global_step‚3199 % _timestamp‚1.7843654918982906e+09 _runtime‚2770.404343499 _step‚259ƒ  '2» - _wandbruntime‚2770  _runtime‚2770UH=à'¼ ÚO % _timestamp‚1.7843654918995268e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3369235542528 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25270922 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25270924 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚484714085318656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5607521 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320861184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29573987 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485594500734625 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12466302 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484714102358016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488083679239680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3369576881664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚484714102358016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488083679239680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29573987 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844909 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3406601122816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482230076305408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25270924 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12466024 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485594143145375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3364083879327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25270922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2770.407798566 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5607518 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6858506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107207390865408 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚488083679239680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29573987 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844909 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107207372253184 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3406601122816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚482230076305408 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25270924 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12466024 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚485594143145375 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3364083879327 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25270922 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 _runtime‚2770.407798566 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 115577856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5607518 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6858506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107207390865408 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚488083320861184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29573987 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚485594500734625 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚96965632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12466302 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚484714102358016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚488083679239680 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3369576881664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0„A•Âê2¾ - _wandbruntime‚2771  _runtime‚2771º·Íê&¿ äM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29593030 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3370589515776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚484861326348288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29583039 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6860549 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110648324168192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107240819623936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482376586977280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚2771.040145412 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3407615920640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485741664870969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484861343387648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871586 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370589515776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843654925340326e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488231574517760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25278554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6860816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25278556 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3365094933049 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871586 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861593 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29583039 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861593 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469812 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5609266 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932903424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3407523156480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12470082 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29593030 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5609263 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6860549 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107240801011712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110648324168192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚107240819623936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚482376586977280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚2771.040145412 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3407615920640 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚485741664870969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚484861343387648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871586 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚482376569937920 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3370589515776 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843654925340326e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚488231574517760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚484861343387648 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25278554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6860816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25278556 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3365094933049 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871586 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861593 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29583039 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861593 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12469812 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5609266 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚488231932903424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3407523156480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12470082 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29593030 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5609263 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6633368730545044 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6723167300224304 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16112840175628662 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6111781001091003 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15890029072761536 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5390754342079163 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5961307287216187 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1412345916032791 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15161658823490143 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01327197439968586 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20848174393177032 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13320204615592957 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987975537776947 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.637631893157959 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8635433912277222 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.589582085609436 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7493575811386108 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35648149251937866 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2680642306804657 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7467344403266907 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.9105002284049988 particle/accuracy_2_of_5‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_4‚0.0907195508480072 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1411563605070114 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.677841067314148 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5394126176834106 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13795994222164154 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_3‚0.9989922046661377 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959957599639893 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4334496557712555 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5631243586540222 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5606133341789246 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06371615827083588 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13274337351322174 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7359222173690796 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.021734412759542465 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5988460183143616 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5750155448913574 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333134651184 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7162736058235168 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9937809705734253 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5160232186317444 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1374068409204483 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14388838410377502 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3149847090244293 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15351751446723938 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7467344403266907 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.589458167552948 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873201251029968 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1412345916032791 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7371565699577332 (loss/val_total_loss‚9.55885345228374 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1408931314945221 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11708205938339233 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843655098473773e+09 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5083824992179871 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00540141761302948 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14989054203033447 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.535461962223053 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9216376543045044 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5483978390693665 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010000340640544891 /particle/accuracy_3_of_4‚0.9938491582870483 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1355370134115219 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6469924449920654 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2007937878370285 +jet/accuracy_1_of_1‚0.26562052965164185 particle/accuracy_1_of_4‚1 _runtime‚2788.354041596 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1389467418193817 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9120371341705322 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5776284337043762 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15473324060440063 particle/accuracy_2_of_2‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5394126176834106 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1401432603597641 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_2_of_4‚0.7622552514076233 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.573565661907196 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5115049481391907 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6164635419845581 _step‚263 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.17592592537403107 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1191544383764267 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6932369470596313 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1550426334142685 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6620141267776489 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6205201745033264 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.12203890830278397 *jet/accuracy_2_of_3‚0.4534325897693634 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939386248588562 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2396179437637329 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7566515803337097 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6843894720077515 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 trainer/global_step‚3199 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1682349443435669 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6170923709869385 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5409846305847168 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6251473426818848 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11708205938339233 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1555052399635315 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7118823528289795 *particle/sensitivity‚0.592271625995636 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.594267725944519 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950833916664124 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.032050471752882004 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.17592592537403107 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14004847407341003 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025354156270623207 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18202514946460724 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25039413571357727 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966475367546082 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4684899151325226 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976009130477905 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 epoch‚7 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948920011520386 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9345056414604187 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4533376693725586 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5348913669586182 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6477354764938354 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.71006840467453 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (jet/accuracy_1_of_4‚0.96027672290802 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3515625 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4457477331161499 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_3‚0.8956273198127747 6validation_average_jet_accuracy‚0.3146894872188568™ˆ2{3È ¶{ ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974848628044128 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7467344403266907 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.589458167552948 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9873201251029968 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965230822563171 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1412345916032791 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_2‚0.7371565699577332 (loss/val_total_loss‚9.55885345228374 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1408931314945221 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11708205938339233 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843655098473773e+09 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5083824992179871 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00540141761302948 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14989054203033447 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.535461962223053 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9216376543045044 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5483978390693665 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010000340640544891 /particle/accuracy_3_of_4‚0.9938491582870483 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1355370134115219 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6469924449920654 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2007937878370285 +jet/accuracy_1_of_1‚0.26562052965164185 particle/accuracy_1_of_4‚1 _runtime‚2788.354041596 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1389467418193817 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9120371341705322 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5776284337043762 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15473324060440063 particle/accuracy_2_of_2‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5394126176834106 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1401432603597641 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_2_of_4‚0.7622552514076233 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.573565661907196 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5115049481391907 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6164635419845581 _step‚263 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.17592592537403107 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1191544383764267 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6932369470596313 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1550426334142685 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6620141267776489 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6205201745033264 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_2_of_2‚0.12203890830278397 *jet/accuracy_2_of_3‚0.4534325897693634 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939386248588562 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2396179437637329 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962049126625061 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7566515803337097 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6843894720077515 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 trainer/global_step‚3199 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,&Purity/0FRt2SRqqt22þ‰)FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1682349443435669 particle/accuracy_2_of_4‚1 'particle/f_score‚0.6170923709869385 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5409846305847168 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6251473426818848 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11708205938339233 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1555052399635315 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7118823528289795 *particle/sensitivity‚0.592271625995636 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.594267725944519 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9950833916664124 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992794394493103 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.032050471752882004 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.17592592537403107 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14004847407341003 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025354156270623207 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18202514946460724 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25039413571357727 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966475367546082 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4684899151325226 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976009130477905 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982566833496094 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_5_of_5‚0 epoch‚7 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948920011520386 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9855887293815613 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9345056414604187 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4533376693725586 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5348913669586182 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6477354764938354 7"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.71006840467453 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 (jet/accuracy_1_of_4‚0.96027672290802 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3515625 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4457477331161499 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_3‚0.8956273198127747 6validation_average_jet_accuracy‚0.3146894872188568 particle/accuracy_1_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15512609481811523 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10427004098892212 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23317770659923553 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10722319036722183 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7391520738601685 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.10619943588972092 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7513983845710754 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5897574424743652 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2132365107536316 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7625472545623779 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964202642440796 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 (particle/accuracy‚0.8421611785888672 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9896620512008667 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5439086556434631 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8615426421165466 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8952076435089111 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3201233446598053 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5928897261619568 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974848628044128 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6633368730545044 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6723167300224304 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16112840175628662 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6111781001091003 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.15890029072761536 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5390754342079163 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5961307287216187 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1412345916032791 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15161658823490143 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01327197439968586 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.20848174393177032 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13320204615592957 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987975537776947 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.637631893157959 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8635433912277222 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.589582085609436 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7493575811386108 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.35648149251937866 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9277234673500061 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2680642306804657 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7467344403266907 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +particle/specificity‚0.9105002284049988 particle/accuracy_2_of_5‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_4‚0.0907195508480072 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1411563605070114 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.677841067314148 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.993180513381958 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5394126176834106 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13795994222164154 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_3‚0.9989922046661377 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959957599639893 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4334496557712555 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5631243586540222 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5606133341789246 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.06371615827083588 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13274337351322174 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7359222173690796 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.021734412759542465 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5988460183143616 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5750155448913574 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2708333134651184 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7162736058235168 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9937809705734253 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5160232186317444 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1374068409204483 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14388838410377502 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3149847090244293 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15351751446723938 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225‡óÃY)›É ‚(: ǃíÒúá÷ï gpu.0.gpu‚16.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚30.0 gpu.0.powerWatts‚91.69' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.337999999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ñ‘¸Ê ‚(:­ ǃíÒ“Ö¢ò% disk.nvme0n1.out‚6221.57275390625 disk.rbd1.in‚0! proc.memory.rssMB‚ 2906.828125 proc.cpu.threads‚17 network.recv‚762957 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3469.9453125 disk.dm-0.in‚ 3469.9453125 disk.rbd0.out‚0" disk.dm-0.out‚6221.57275390625 disk.sda.in‚0$ memory_percent‚96.30053043365479 cpu‚25.968296953825746# disk./.usageGB‚589.908504486084 disk.dm-1.in‚0 disk.dm-1.out‚ 60.9609375 disk.sda2.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.58925470226446 disk.rbd0.in‚0* proc.memory.percent‚17.741870880126953 disk.rbd1.out‚7.5" disk.nvme0c0n1.in‚ 3469.9453125 network.sent‚ 5.11286e+06 disk.sda2.out‚0 disk.sda.out‚ 60.9609375 disk.sda1.in‚0 disk.sda3.out‚ 60.9609375' disk.nvme0c0n1.out‚6221.57275390625( proc.memory.availableMB‚ 606.12109375ûgDšžË ‚(:“ ÖƒíÒ«»™í gpu.0.gpu‚91.0 gpu.0.memory‚31+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚256.933' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.386599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕS~¶Ì ‚(:« ÖƒíҪ׶ï network.recv‚766954 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 740.26171875 disk.sda3.in‚0 disk.dm-0.in‚ 3469.9453125 disk.sda2.out‚0$ memory_percent‚95.48180103302002 proc.cpu.threads‚17) disk./.usagePercent‚39.58918890832817" disk.nvme0c0n1.in‚ 3469.9453125 disk.sda3.out‚ 61.171875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2771.65234375 cpu‚25.965869027501277$ disk./.usageGB‚589.9075241088867& disk.nvme0c0n1.out‚6493.5908203125 disk.sda.in‚0 disk.sda.out‚ 61.171875 disk.sda1.out‚0 disk.nvme0n1.in‚ 3469.9453125 network.sent‚ 5.157221e+06 disk.rbd1.in‚0 disk.dm-1.in‚0* proc.memory.percent‚16.916823387145996 disk.drbd2165.out‚0 disk.dm-1.out‚ 61.171875$ disk.nvme0n1.out‚6493.5908203125! disk.dm-0.out‚6493.5908203125 disk.sda2.in‚0 disk.rbd1.out‚7.5i<“Í ‚(:ˆ åƒíÒ‹Å…ï gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üs9»Î ‚(:° åƒíÒ¯“Âñ) disk./.usagePercent‚39.58920426878022 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚6719.826171875 disk.dm-0.in‚ 3469.9453125 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 61.59765625# proc.memory.rssMB‚ 2771.65234375$ disk./.usageGB‚589.9077529907227" disk.nvme0c0n1.in‚ 3469.9453125 proc.cpu.threads‚17 network.sent‚ 5.164155e+06 disk.rbd1.out‚7.5625 disk.sda3.out‚ 61.59765625' proc.memory.availableMB‚ 740.2578125 cpu‚25.964339959143103 disk.sda.out‚ 61.59765625 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.recv‚770573 disk.dm-0.out‚6719.826171875 disk.sda2.in‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚6719.826171875 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3469.9453125$ memory_percent‚95.48182487487793* proc.memory.percent‚16.916823387145996±6t«2Ï - _wandbruntime‚2830  _runtime‚2830s’àÆÐ À & lr-AdamW/pg1‚0.0005522037677189803 & lr-AdamW/pg2‚0.0005522037677189803 trainer/global_step‚3249 $ _timestamp‚1.784365552248832e+09 _runtime‚2830.755695281 _step‚264j‘©ËÑ Å & lr-AdamW/pg1‚0.0005522037677189803 & lr-AdamW/pg2‚0.0005522037677189803 trainer/global_step‚3249 $ _timestamp‚1.784365552248832e+09 _runtime‚2830.755695281 _step‚264ˆN‡M2Ò - _wandbruntime‚2830  _runtime‚2830¢°óß'Ó ÙO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112370608677376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664252416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494005703450624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12692021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108894722279424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3432420091469 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30108743 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367703040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30098766 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6994730 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367703040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108894740891648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3432760641359 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2830.756090147 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30098766 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12691753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494005720489984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚494005703450624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3475886397952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25773188 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5697291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888Å ÞSÔ ÞO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6994730 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚497443367703040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108894740891648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3432760641359 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2830.756090147 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30098766 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12691753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚494005720489984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚494005703450624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3475886397952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25773188 UGDeviceStatsMonitor.onv6±/‘ _train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5697291 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112370608677376 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3437664252416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚494005703450624 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12692021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108894722279424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3432420091469 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30108743 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚497443367703040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30098766 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176‰eÔE2Õ - _wandbruntime‚2831  _runtime‚2831´œÊHê&Ö äM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3439018225664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚3249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30107818 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25780820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12695551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494152944480256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112405167401984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚495044197890791 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676879360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495044555480153 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2831.386275922 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888638 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476909181952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476997751808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107818 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112405060219904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491610766745600 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30117809 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚491610783784960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3433431145191 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚494152961519616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3438676879360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621359616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6996791 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979745280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108928169650176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780820¸SJ®ï&× éM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3476909181952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898631 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3476997751808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30107818 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112405060219904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491610766745600 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30117809 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚491610783784960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3433431145191 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚494152961519616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3438676879360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚497591621359616 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6996791 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888638 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚497591979745280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108928169650176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25780820 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3439018225664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚3249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30107818 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25780820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12695551 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚494152944480256 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112405167401984 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚495044197890791 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3438676879360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚495044555480153 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2831.386275922 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888638 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77Šç Á;2Ø - _wandbruntime‚2831  _runtime‚2831žŸEÙ € _runtime‚2831.387415554 /loss/FRt1/assignment_loss‚2.083881167644793 0loss/FRt2/assignment_loss‚1.0837931492157626 2loss/SRqqt2/assignment_loss‚0.5542109240996903 1loss/SRqqt1/detection_loss‚1.0637833135990569 %loss/total_loss‚9.819719287975243 _step‚267 2loss/FBt1/assignment_loss‚0.006020974224960823 -loss/FRt1/detection_loss‚0.62442583787013 /loss/FBt2/detection_loss‚0.8634243421017741 trainer/global_step‚3249 % _timestamp‚1.7843655528817437e+09 1loss/SRqqt2/detection_loss‚0.6816422389582939 2loss/SRqqt1/assignment_loss‚0.7164252305654093 2loss/FBt2/assignment_loss‚0.023986964026909856 /loss/FRt2/detection_loss‚1.5562943518810164 /loss/FBt1/detection_loss‚0.5618307937874453 epoch‚8o[œß Ú … 1loss/SRqqt2/detection_loss‚0.6816422389582939 2loss/SRqqt1/assignment_loss‚0.7164252305654093 2loss/FBt2/assignment_loss‚0.023986964026909856 /loss/FRt2/detection_loss‚1.5562943518810164 /loss/FBt1/detection_loss‚0.5618307937874453 epoch‚8 _runtime‚2831.387415554 /loss/FRt1/assignment_loss‚2.083881167644793 0loss/FRt2/assignment_loss‚1.0837931492157626 2loss/SRqqt2/assignment_loss‚0.5542109240996903 1loss/SRqqt1/detection_loss‚1.0637833135990569 %loss/total_loss‚9.819719287975243 _step‚267 2loss/FBt1/assignment_loss‚0.006020974224960823 -loss/FRt1/detection_loss‚0.62442583787013 /loss/FBt2/detection_loss‚0.8634243421017741 trainer/global_step‚3249 % _timestamp‚1.7843655528817437e+09‹:WC·žÛ ‚(:“ ôƒíÒ¾Í÷ï gpu.0.gpu‚45.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U:d»Ü ‚(:° ôƒíÒÂÎœò disk.sda3.out‚ 61.89453125 disk.dm-1.out‚ 61.89453125 disk.sda2.in‚0 disk.sda3.in‚0 network.recv‚773923! disk.nvme0n1.in‚ 3552.89453125% disk.nvme0c0n1.out‚6723.697265625* proc.memory.percent‚16.916823387145996 cpu‚25.963027099996783 disk.drbd2165.out‚0 disk.dm-0.out‚6723.697265625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚6723.697265625 disk.dm-0.in‚ 3552.89453125 disk.sda2.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2771.65234375 network.sent‚ 5.18016e+06$ disk./.usageGB‚589.9079437255859 disk.drbd2165.in‚0 disk.rbd1.out‚7.5625# disk.nvme0c0n1.in‚ 3552.89453125 disk.sda1.in‚0) disk./.usagePercent‚39.58921706915694 disk.dm-1.in‚0 disk.sda.out‚ 61.89453125$ memory_percent‚95.48168182373047 proc.cpu.threads‚17 disk.sda1.out‚0% proc.memory.availableMB‚ 740.28125ÿþ†“Ý ‚(:ˆ ƒ„íÒº´î gpu.0.gpu‚85.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚239.187' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.8374 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CüVBÃÞ ‚(:¸ ƒ„íÒ…¦®ï* proc.memory.percent‚16.916823387145996# disk.nvme0c0n1.in‚ 3552.89453125 disk.sda3.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚6730.3212890625$ memory_percent‚95.48366069793701 disk.sda2.in‚0 disk.sda.out‚ 62.16015625 disk.dm-1.out‚ 62.16015625 network.recv‚778679$ disk./.usageGB‚589.9079246520996 disk.sda2.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚6730.3212890625* disk./.usagePercent‚39.589215789119265 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd1.out‚7.6875& disk.nvme0c0n1.out‚6730.3212890625 disk.sda3.out‚ 62.16015625 disk.dm-1.in‚0 disk.drbd2165.out‚0 cpu‚25.961534599617746 proc.cpu.threads‚17 disk.dm-0.in‚ 3552.89453125 network.sent‚ 5.230044e+06 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 3552.89453125( proc.memory.availableMB‚ 739.95703125# proc.memory.rssMB‚ 2771.65234375ÑÉrÁ“ß ‚(:ˆ ’„íÒ¸•ï gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÜbO»à ‚(:° ’„íÒú‹³ñ' disk.nvme0c0n1.out‚6734.78271484375 disk.dm-1.out‚62.46875 cpu‚25.960138921706445 disk.sda1.in‚0% disk.nvme0n1.out‚6734.78271484375 disk.sda.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0' proc.memory.availableMB‚ 740.2109375 network.sent‚ 5.234871e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚62.46875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2771.65234375$ memory_percent‚95.48211097717285 network.recv‚781832* disk./.usagePercent‚39.589210412961044 disk.dm-0.in‚ 3552.89453125# disk.nvme0c0n1.in‚ 3552.89453125* proc.memory.percent‚16.916823387145996" disk.dm-0.out‚6734.78271484375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 3552.89453125 disk.sda3.in‚0 disk.sda.out‚62.46875# disk./.usageGB‚589.907844543457 disk.rbd1.out‚7.6875“ÁZ2á - _wandbruntime‚2871  _runtime‚2871ÁÞê2Çâ Á _runtime‚2871.757862948 _step‚268 & lr-AdamW/pg1‚0.0005511669965750276 & lr-AdamW/pg2‚0.0005511669965750276 trainer/global_step‚3299 % _timestamp‚1.7843655932508419e+09û™˜RÌã Æ & lr-AdamW/pg2‚0.0005511669965750276 trainer/global_step‚3299 % _timestamp‚1.7843655932508419e+09 _runtime‚2871.757862948 _step‚268 & lr-AdamW/pg1‚0.0005511669965750276ŒXñÎ2ä - _wandbruntime‚2871  _runtime‚2871i¡ }¯ å ÚO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883781 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7099240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚498789806706688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498789789667328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3488359754240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3527153756672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502273183834955 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚3299 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114686464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5784541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504856473064960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3488359754240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30563195 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501367754932224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114093313962496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3483377128267 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154788 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26154788 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843655932516007e+09 _runtime‚2871.757973808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717985 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701093376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5784538 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438.å'æ ßO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12883781 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7099240 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚498789806706688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚498789789667328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3488359754240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3527153756672 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502273183834955 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 trainer/global_step‚3299 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚504856114686464 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5784541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚504856473064960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3488359754240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30563195 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501367754932224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114093313962496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3483377128267 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26154788 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501367771971584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26154788 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 % _timestamp‚1.7843655932516007e+09 _runtime‚2871.757973808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717985 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3488701093376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5784538 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 3Ȳ2ç - _wandbruntime‚2872  _runtime‚2872jRƒê&è äM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724690 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843655938797317e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734683 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724690 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498936300339200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5786283 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30562270 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501514995961856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30562270 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3489372381184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12887588 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110599607576576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502420705560661 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3484047632099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30572261 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚501515013001216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726728704 _runtime‚2872.385860943 _step‚270 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26162422 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3489713727488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26162420 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110599588964352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26162420 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3489372381184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3489713727488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7101028 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505004726728704 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12887311 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7101302 B«é éM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚498936300339200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5786283 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30562270 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚501514995961856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30562270 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3489372381184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12887588 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚110599607576576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚501515013001216 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚502420705560661 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3484047632099 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30572261 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚501515013001216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚505004726728704 _runtime‚2872.385860943 _step‚270 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26162422 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3489713727488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26162420 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110599588964352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26162420 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3489372381184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3489713727488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7101028 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505004726728704 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12887311 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7101302 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843655938797317e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734683 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3484388182101 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56734683 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚501514995961856 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724690 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Ž‰Ç„Ì2ê - _wandbruntime‚2872  _runtime‚2872—•ž· ë „ 2loss/SRqqt1/assignment_loss‚0.5844249753546923 3loss/FBt2/assignment_loss‚0.0006745951098673686 /loss/FRt1/detection_loss‚0.5620804729792215 1loss/SRqqt1/detection_loss‚1.0766188065142577 0loss/FRt1/assignment_loss‚2.1484757593371033 0loss/FRt2/assignment_loss‚1.1674241108271097 2loss/SRqqt2/assignment_loss‚0.5566609742604393 /loss/FRt2/detection_loss‚1.5015865472711514 1loss/SRqqt2/detection_loss‚0.6890093508037667 0loss/FBt1/detection_loss‚0.47448322550371813 epoch‚8 trainer/global_step‚3299 1loss/FBt1/assignment_loss‚0.03833085198935038 .loss/FBt2/detection_loss‚1.335400117598474 &loss/total_loss‚10.135169787549152 % _timestamp‚1.7843655938814132e+09 _runtime‚2872.387406922 _step‚271nTáì ‰ epoch‚8 trainer/global_step‚3299 1loss/FBt1/assignment_loss‚0.03833085198935038 .loss/FBt2/detection_loss‚1.335400117598474 &loss/total_loss‚10.135169787549152 % _timestamp‚1.7843655938814132e+09 _runtime‚2872.387406922 _step‚271 2loss/SRqqt1/assignment_loss‚0.5844249753546923 3loss/FBt2/assignment_loss‚0.0006745951098673686 /loss/FRt1/detection_loss‚0.5620804729792215 1loss/SRqqt1/detection_loss‚1.0766188065142577 0loss/FRt1/assignment_loss‚2.1484757593371033 0loss/FRt2/assignment_loss‚1.1674241108271097 2loss/SRqqt2/assignment_loss‚0.5566609742604393 /loss/FRt2/detection_loss‚1.5015865472711514 1loss/SRqqt2/detection_loss‚0.6890093508037667 0loss/FBt1/detection_loss‚0.47448322550371813Õ§ã“í ‚(:ˆ ¡„íÒ„×ï gpu.0.gpu‚44.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚281.999' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Ö%·î ‚(:¬ ¡„íÒþ¸ñ" disk.nvme0n1.out‚ 6741.17578125 disk.dm-0.in‚ 3552.89453125 network.sent‚ 5.252985e+06 disk.dm-1.in‚0! disk.nvme0n1.in‚ 3552.89453125 disk.sda.out‚ 62.7734375$ disk./.usageGB‚589.9078140258789 disk.dm-0.out‚ 6741.17578125 disk.sda2.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 6741.17578125$ memory_percent‚95.48532962799072 cpu‚25.958880917734074 network.recv‚785703 disk.dm-1.out‚ 62.7734375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚7.75 disk.sda1.in‚0# proc.memory.rssMB‚ 2771.65234375) disk./.usagePercent‚39.58920836490077 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda.in‚0( proc.memory.availableMB‚ 739.68359375 disk.sda3.out‚ 62.7734375 disk.drbd2165.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 3552.89453125 disk.sda1.out‚0* proc.memory.percent‚16.916823387145996 proc.cpu.threads‚17!ÿö+žï ‚(:“ °„íÒ¶¸¨í gpu.0.gpu‚85.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚224.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.817800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rç¹ð ‚(:® °„íÒðâàï disk.sda.in‚0 disk.nvme0n1.in‚ 3660.3359375# proc.memory.rssMB‚ 2771.65234375* proc.memory.percent‚16.916823387145996% disk.nvme0n1.out‚6744.52685546875" disk.dm-0.out‚6744.52685546875# memory_percent‚95.4840898513794 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 63.203125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚7.75" disk.nvme0c0n1.in‚ 3660.3359375' disk.nvme0c0n1.out‚6744.52685546875 disk.sda.out‚ 63.203125 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚39.58920478079529 disk.sda1.out‚0 disk.dm-1.out‚ 63.203125( proc.memory.availableMB‚ 739.88671875 cpu‚25.957496471282955 network.recv‚789767$ disk./.usageGB‚589.9077606201172 disk.dm-0.in‚ 3660.3359375 network.sent‚ 5.302775e+06ž²Ïñ ‚(:’ ¿„íÒÖŸªí gpu.0.gpu‚88.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05À ¹ò ‚(:® ¿„íÒÙ̸ï cpu‚25.956101096979864 disk.nvme0n1.in‚ 3660.3359375 disk.sda2.out‚0$ disk./.usageGB‚589.9077682495117 disk.drbd2165.out‚0 disk.rbd0.out‚0 network.recv‚793128 disk.sda3.in‚0# disk.nvme0n1.out‚6749.931640625 disk.dm-0.in‚ 3660.3359375$ memory_percent‚95.48470973968506# proc.memory.rssMB‚ 2771.65234375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3660.3359375 disk.sda1.out‚0* proc.memory.percent‚16.916823387145996 disk.sda3.out‚ 63.4609375 disk.rbd0.in‚0 disk.rbd1.out‚7.8125% disk.nvme0c0n1.out‚6749.931640625( proc.memory.availableMB‚ 739.78515625) disk./.usagePercent‚39.58920529281036 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.out‚ 63.4609375 proc.cpu.threads‚17 network.sent‚ 5.307697e+06 disk.dm-1.out‚ 63.4609375 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.out‚6749.931640625 disk.sda1.in‚0Nõ Ì2ó - _wandbruntime‚2912  _runtime‚2912‡…kõÆô À & lr-AdamW/pg1‚0.0005501135161204187 & lr-AdamW/pg2‚0.0005501135161204187 trainer/global_step‚3349 $ _timestamp‚1.784365634072203e+09 _runtime‚2912.578196647 _step‚272]‚¿Ëõ Å & lr-AdamW/pg2‚0.0005501135161204187 trainer/global_step‚3349 $ _timestamp‚1.784365634072203e+09 _runtime‚2912.578196647 _step‚272 & lr-AdamW/pg1‚0.0005501135161204187Òl¢2ö - _wandbruntime‚2912  _runtime‚2912Piùà'÷ ÚO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26536388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396595200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚512268861669888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396595200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3349 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861669888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3539055256064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31017647 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115816087940608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31007670 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26536390 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115815966071296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7203952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506115340300288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112237598132224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26536390 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚512269220048384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚508729806413824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26536388 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121869312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112237616744448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚508729806413824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3539396595200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚512268861669888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3539396595200 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚3349 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚512268861669888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3539055256064 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31017647 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115816087940608 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31007670 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26536390 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚115815966071296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5871788 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7203952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚506115340300288 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616‘çËOº2ù - _wandbruntime‚2913  _runtime‚2913"i7~ú äM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3535004669009 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5873533 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067883008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31016722 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13079860 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚506261833932800 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560742 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579387937792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚274 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3540067883008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚508877064482816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7206044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112271045502976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 trainer/global_step‚3349 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57570735 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540409229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509796855641169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115850414828544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509796498051807 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570735 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579480701952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3534664119007 _runtime‚2913.210933665 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31026713 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115850526204928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112271026890752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5873536 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7206324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚512417473712128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026713 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13079577 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0[Áâûï&û éM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31016722 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5873533 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3540067883008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31016722 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13079860 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚506261833932800 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560742 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579387937792 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚274 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3540067883008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚508877064482816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7206044 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚112271045502976 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 trainer/global_step‚3349 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57570735 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3540409229312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚509796855641169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115850414828544 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚509796498051807 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570735 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3579480701952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3534664119007 _runtime‚2913.210933665 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚508877047443456 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31026713 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115850526204928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112271026890752 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5873536 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7206324 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚512417473712128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26544022 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31026713 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13079577 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26544020 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3535004669009’{Räe2ü - _wandbruntime‚2913  _runtime‚2913ÀêaÜÿý ù /loss/FRt1/assignment_loss‚2.166226496775183 2loss/SRqqt2/assignment_loss‚0.5735094115088256 /loss/FRt1/detection_loss‚0.5609907464889625 -loss/FRt2/detection_loss‚1.47621435201056 .loss/FBt2/detection_loss‚1.163029578225366 trainer/global_step‚3349 _runtime‚2913.212104333 1loss/FBt2/assignment_loss‚0.00257266865937003 1loss/SRqqt1/detection_loss‚0.9440286310385725 &loss/total_loss‚10.014880807544028 epoch‚8 0loss/FRt2/assignment_loss‚1.1161827001783808 2loss/SRqqt1/assignment_loss‚0.6937780541907962 0loss/SRqqt2/detection_loss‚0.688711017295767 /loss/FBt1/detection_loss‚0.5922160287750245 $ _timestamp‚1.784365634706428e+09 .loss/FBt1/assignment_loss‚0.03742112239722 _step‚275ö#•Aþ þ .loss/FBt1/assignment_loss‚0.03742112239722 _step‚275 /loss/FRt1/assignment_loss‚2.166226496775183 2loss/SRqqt2/assignment_loss‚0.5735094115088256 /loss/FRt1/detection_loss‚0.5609907464889625 -loss/FRt2/detection_loss‚1.47621435201056 .loss/FBt2/detection_loss‚1.163029578225366 trainer/global_step‚3349 _runtime‚2913.212104333 1loss/FBt2/assignment_loss‚0.00257266865937003 1loss/SRqqt1/detection_loss‚0.9440286310385725 &loss/total_loss‚10.014880807544028 epoch‚8 0loss/FRt2/assignment_loss‚1.1161827001783808 2loss/SRqqt1/assignment_loss‚0.6937780541907962 0loss/SRqqt2/detection_loss‚0.688711017295767 /loss/FBt1/detection_loss‚0.5922160287750245 $ _timestamp‚1.784365634706428e+09“êx¥Ž’ÿ ‚(:‡ ΄íÒê—«ï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a0Ôx¼€‚(:± ΄íÒÍ•¸ñ disk.sda2.in‚0" disk.dm-0.out‚6753.19189453125* proc.memory.percent‚16.916823387145996 disk.sda1.in‚0 disk.rbd0.out‚0$ memory_percent‚95.48637866973877( proc.memory.availableMB‚ 739.51171875 network.sent‚ 5.325808e+06 network.recv‚796561 disk.sda2.out‚0 disk.dm-0.in‚ 3660.3359375# proc.memory.rssMB‚ 2771.65234375 proc.cpu.threads‚17# disk./.usageGB‚589.907772064209 disk.sda3.out‚ 63.703125 disk.drbd2165.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚39.589205548817894 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 3660.3359375' disk.nvme0c0n1.out‚6753.19189453125% disk.nvme0n1.out‚6753.19189453125 disk.rbd1.out‚7.8125 disk.sda.in‚0 disk.sda.out‚ 63.703125 disk.sda3.in‚0 disk.nvme0n1.in‚ 3660.3359375 cpu‚25.954846971062473 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 63.703125 disk.drbd2165.out‚0ÌéW©œ‚(:‘ Ý„íÒŒ­”í gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚291.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.383799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B'µ‚‚(:ª Ý„íÒ°¿—ï network.sent‚ 5.37569e+06$ disk./.usageGB‚589.9077911376953 disk.rbd1.out‚7.875 disk.drbd2165.in‚0) disk./.usagePercent‚39.58920682885556 disk.rbd1.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 3660.3359375% disk.nvme0c0n1.out‚6756.919921875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2771.65234375 cpu‚25.953430006707546 disk.nvme0n1.in‚ 3660.3359375# disk.nvme0n1.out‚6756.919921875$ memory_percent‚95.48482894897461& proc.memory.availableMB‚ 739.765625* proc.memory.percent‚16.916823387145996 disk.dm-0.out‚6756.919921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 64.1328125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 64.1328125 network.recv‚801113 disk.dm-0.in‚ 3660.3359375 disk.sda.out‚ 64.1328125´ó“2ƒ- _wandbruntime‚2953  _runtime‚2953õnµÜÇ„Á trainer/global_step‚3399 % _timestamp‚1.7843656750718615e+09 _runtime‚2953.577918516 _step‚276 & lr-AdamW/pg1‚0.0005490433940182345 & lr-AdamW/pg2‚0.0005490433940182345[ŠÒÌ…Æ & lr-AdamW/pg2‚0.0005490433940182345 trainer/global_step‚3399 % _timestamp‚1.7843656750718615e+09 _runtime‚2953.577918516 _step‚276 & lr-AdamW/pg1‚0.0005490433940182345”(ŠXL2†- _wandbruntime‚2953  _runtime‚2953.k§m &‡ÛO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517025483995971 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚513440873893888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517025126406721 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚516091874934784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚2953.579216146 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13267125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117538604430336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103257088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3589750757888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843656750730526e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608653312 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3590092097024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5959038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26917990 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26917988 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513440856854528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31472099 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519681967031808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26917990 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚516091874934784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519681608653312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚516091857895424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26917988 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308084 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31462122 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31462122 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13266841 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121869312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117538482561024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629446502400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3590092097024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚3399 _step‚277 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3584269552193 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31472099 WIDevic_¦“×eStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3589750757888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7307803 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0Ï'Jæ'ˆàO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3589750757888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7307803 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚517025483995971 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚513440873893888 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚517025126406721 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚516091874934784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _runtime‚2953.579216146 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13267125 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117538604430336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 103257088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3589750757888 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843656750730526e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚519681608653312 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3590092097024 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5959038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26917990 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26917988 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚513440856854528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31472099 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚519681967031808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26917990 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚516091874934784 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚519681608653312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚516091857895424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26917988 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7308084 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31462122 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31462122 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13266841 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 121869312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117538482561024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3629446502400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3590092097024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 trainer/global_step‚3399 _step‚277 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3584269552193 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31472099 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888•ýõˆ2‰- _wandbruntime‚2954  _runtime‚29542l\ë&ŠåM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26925620 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31471174 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516239098925056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519829862309888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155917 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31481165 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396794 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚278 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3590763384832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3591104731136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406787 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚3399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26925622 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516239098925056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13270644 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517172648132315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3590763384832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519830220695552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519830220695552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13270927 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104731136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117572929425920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113942483429376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517173005721677 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513587367526400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113942464817152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843656757085712e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚2954.214777507 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508Íìð&‹êM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31471174 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26925620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3585280605915 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚516239115964416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3630464608768 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚3399 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26925622 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚516239098925056 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚516239115964416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13270644 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚517172648132315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3590763384832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚519830220695552 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5960786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚519830220695552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13270927 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3591104731136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117572929425920 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113942483429376 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚517173005721677 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚513587367526400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113942464817152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 % _timestamp‚1.7843656757085712e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚2954.214777507 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26925620 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31471174 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516239098925056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26925622 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚519829862309888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3585621155917 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31481165 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396794 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚278 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3590763384832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3591104731136 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406787 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31916574 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916574 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226141 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115580473985024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31926551 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27299590 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520766390448128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6046291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119260925039104 _step‚281 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355636736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13458230 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523453926416384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7411939 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7411676 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524401276487229 trainer/global_step‚3449 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787598848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3640787598848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216162 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27299588 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6046288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27299588 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31916574 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31916574 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 117675008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59226141 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚523453909377024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115580473985024 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31926551 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27299590 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚520766390448128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6046291 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119260925039104 _step‚281 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚527094355636736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13458230 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚523453926416384 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7411939 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7411676 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚524401276487229 trainer/global_step‚3449 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3640787598848 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3640787598848 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59216162 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚99062784 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27299588 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6046288 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27299588 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27307220 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3681473838080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 % _timestamp‚1.7843657166664734e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925626 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3681568699392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3636237642825 _step‚282 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚527242967678976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119295376581632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚527242609293312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242839 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115613921355776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚527242967678976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27307222 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242839 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520912901120000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3641800232960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13461732 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3635897092823 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚523601167446016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6048033 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚524549155802185 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27307222 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27307220 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6048036 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27307220 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3681473838080 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232846 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 % _timestamp‚1.7843657166664734e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31925626 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3681568699392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3636237642825 _step‚282 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚527242967678976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232846 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119295376581632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚527242609293312 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242839 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115613921355776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚527242967678976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27307222 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242839 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚520912901120000 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3641800232960 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13461732 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0šjAþƒ2ž- _wandbruntime‚2995  _runtime‚2995‹ Ÿƒ /loss/FRt1/detection_loss‚0.5550680916513719 1loss/SRqqt2/detection_loss‚0.6740340722884017 0loss/FRt2/assignment_loss‚1.1272305113928658 1loss/SRqqt1/detection_loss‚1.0196047615873711 trainer/global_step‚3449 % _timestamp‚1.7843657166681337e+09 _step‚283 /loss/FRt1/assignment_loss‚2.140758998144992 0loss/FBt1/detection_loss‚0.38843597372494687 %loss/total_loss‚9.706119346337857 epoch‚8 _runtime‚2995.173848368 2loss/SRqqt1/assignment_loss‚0.5628178551780062 2loss/SRqqt2/assignment_loss‚0.5983394315369772 /loss/FRt2/detection_loss‚1.4741590887783285 /loss/FBt2/detection_loss‚1.1540134909522275 2loss/FBt1/assignment_loss‚0.008618361204182792 2loss/FBt2/assignment_loss‚0.003038709898185395[ÊÈn ˆ 2loss/SRqqt1/assignment_loss‚0.5628178551780062 2loss/SRqqt2/assignment_loss‚0.5983394315369772 /loss/FRt2/detection_loss‚1.4741590887783285 /loss/FBt2/detection_loss‚1.1540134909522275 2loss/FBt1/assignment_loss‚0.008618361204182792 2loss/FBt2/assignment_loss‚0.003038709898185395 /loss/FRt1/detection_loss‚0.5550680916513719 1loss/SRqqt2/detection_loss‚0.6740340722884017 0loss/FRt2/assignment_loss‚1.1272305113928658 1loss/SRqqt1/detection_loss‚1.0196047615873711 trainer/global_step‚3449 % _timestamp‚1.7843657166681337e+09 _step‚283 /loss/FRt1/assignment_loss‚2.140758998144992 0loss/FBt1/detection_loss‚0.38843597372494687 %loss/total_loss‚9.706119346337857 epoch‚8 _runtime‚2995.173848368›Ü™CEž¡‚(:“ ™…íÒ–Åï gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚255.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.068599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EEÞ½¢‚(:² ™…íÒÿÉÈð disk.nvme0n1.in‚ 3743.1171875$ memory_percent‚95.48368453979492 network.recv‚816236) disk./.usagePercent‚39.58919146840351% disk.nvme0n1.out‚6773.81787109375 disk.sda1.out‚0# proc.memory.rssMB‚ 2771.65234375 disk.sda3.in‚0 disk.sda.out‚ 65.4296875 proc.cpu.threads‚17 network.sent‚ 5.466374e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 65.4296875 disk.dm-0.in‚ 3743.1171875 disk.rbd1.in‚0 disk.rbd1.out‚8.0625" disk.nvme0c0n1.in‚ 3743.1171875 disk.sda.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚6773.81787109375' disk.nvme0c0n1.out‚6773.81787109375& proc.memory.availableMB‚ 739.953125 cpu‚25.948212959102104$ disk./.usageGB‚589.9075622558594 disk.dm-1.out‚ 65.4296875 disk.drbd2165.in‚0* proc.memory.percent‚16.916823387145996 disk.rbd0.in‚0fo ž£‚(:“ ¨…íÒÈç¼ï gpu.0.gpu‚92.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒlĤ‚(:¹ ¨…íÒä›Öð disk.rbd0.out‚0" disk.dm-0.out‚6777.21240234375 disk.rbd1.out‚8.0625 network.sent‚ 5.518231e+06 disk.sda2.out‚0 disk.sda1.in‚0 cpu‚25.94690394780805) disk./.usagePercent‚39.58920606083296 disk.sda3.out‚ 65.81640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.916823387145996# disk.nvme0c0n1.in‚ 3856.41796875$ memory_percent‚95.48628330230713 network.recv‚820656 disk.dm-1.out‚ 65.81640625 disk.dm-0.in‚ 3856.41796875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚6777.21240234375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚6777.21240234375 disk.sda.out‚ 65.81640625# proc.memory.rssMB‚ 2771.65234375! disk.nvme0n1.in‚ 3856.41796875 disk.sda2.in‚0( proc.memory.availableMB‚ 739.52734375 disk.sda.in‚0$ disk./.usageGB‚589.9077796936035 disk.dm-1.in‚0¦ºæ“¥‚(:ˆ ·…íÒÍ÷þí gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†Ö"%æ‚(:¸ ·…íÒíìî" disk.dm-0.out‚6782.26025390625 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 66.13671875% disk.nvme0n1.out‚6782.26025390625 network.recv‚824323 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 3856.41796875 disk.sda.in‚0 network.sent‚ 5.521301e+06) disk./.usagePercent‚39.58843189404942 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 66.13671875 disk.sda1.in‚0! disk.nvme0n1.in‚ 3856.41796875' proc.memory.availableMB‚ 739.6484375 proc.cpu.threads‚17 disk.sda3.out‚ 66.13671875 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚8.125# proc.memory.rssMB‚ 2771.65234375 disk.dm-0.in‚ 3856.41796875' disk.nvme0c0n1.out‚6782.26025390625$ memory_percent‚95.48554420471191* proc.memory.percent‚16.916823387145996 cpu‚25.945805805418864$ disk./.usageGB‚589.8962440490723 disk.drbd2165.out‚0Þu\f2§- _wandbruntime‚3035  _runtime‚3035\s¹VƨÀ & lr-AdamW/pg2‚0.0005468535008633597 trainer/global_step‚3499 $ _timestamp‚1.784365756828458e+09 _runtime‚3035.334411871 _step‚284 & lr-AdamW/pg1‚0.0005468535008633597ÌÊ€™Ë©Å _runtime‚3035.334411871 _step‚284 & lr-AdamW/pg1‚0.0005468535008633597 & lr-AdamW/pg2‚0.0005468535008633597 trainer/global_step‚3499 $ _timestamp‚1.784365756828458e+09œ\2ª- _wandbruntime‚3035  _runtime‚3035Y–‡öá'«ÛO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27681188 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062193 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27681188 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843657568295617e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052214 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3731622030336 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3691483100672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531777784156987 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102620160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3691483100672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120983432781824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6133538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120983552553984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528091941081088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117251930523648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚3035.335793078 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3691141761536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6133541 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534507460998656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27681190 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117251911911424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27681190 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534507102620160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3685502526009 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13650151 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚3499 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13649870 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚534507460998656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27681190 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117251911911424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27681190 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚534507102620160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3685502526009 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13650151 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 trainer/global_step‚3499 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13649870 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530815960858624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530815977897984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27681188 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062193 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27681188 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843657568295617e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052214 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3731622030336 C:DeviceS³Ý)FtatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3691483100672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531777784156987 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚534507102620160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3691483100672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120983432781824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6133538 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120983552553984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚528091941081088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚117251930523648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 _runtime‚3035.335793078 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3691141761536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6133541 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068898 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534655714662400 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6135283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚530963201888256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 trainer/global_step‚3499 _runtime‚3035.969994088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32390069 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32380078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068898 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078891 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7518660 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528238434713600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3686513579731 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32380078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3732641867264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13653946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6135286 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117285359282176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27688820 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121018001149440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531924948293331 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3686854129733 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3732555394560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚286 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13653691UÞÃEï&¯éM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843657574638748e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3692495734784 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078891 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7518660 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚528238434713600 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3686513579731 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27688820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32380078 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3732641867264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13653946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6135286 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚117285359282176 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27688820 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121018001149440 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚531924948293331 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚528238451752960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3686854129733 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3732555394560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚286 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117285340669952 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13653691 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60068898 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚534655714662400 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6135283 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚530963201888256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 trainer/global_step‚3499 _runtime‚3035.969994088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32390069 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32380078 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068898 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3736119012917 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539153934237495 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541920207982080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207982080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3741837263360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13842079 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚535417457635328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3782665640448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539153576648245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28062788 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7621288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚118923368450048 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849603584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28062790 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541919849603584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3736459562807 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32835455 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60888266 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28062790 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782570771968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚289 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538178029379584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3742178602496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚535417474674688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚3076.508881715 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825478 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32825478 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122705920609792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚3549 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541919849603584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28062790 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541919849603584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3736459562807 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32835455 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6220791 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60888266 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28062790 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782570771968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 _step‚289 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚538178029379584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3742178602496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚535417474674688 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚3076.508881715 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32825478 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32825478 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122705920609792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538178012340224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚3549 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28062788 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3736119012917 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚539153934237495 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541920207982080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚541920207982080 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3741837263360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13842079 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚535417457635328 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3782665640448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚539153576648245 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269¡¢Q 2¿- _wandbruntime‚3077  _runtime‚3077ªIJøê&ÀäM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28070420 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122740476993536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚542068461645824 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚538325270409216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963201 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3783679784960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843657986321776e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6222536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚290 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚538325270409216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚535563968307200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914943 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3737470616641 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28070420 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60904950 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13845620 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118956778596352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚3077.138342631 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28070422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7623339 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914943 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3783587020800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3743191236608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539301098373839 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7623087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914943 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28070422 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3783587020800 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3743191236608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚539301098373839 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7623087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚539301455963201 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3783679784960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 % _timestamp‚1.7843657986321776e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6222536 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _step‚290 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚538325270409216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚535563968307200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914943 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3737470616641 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28070420 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60904950 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13845620 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚118956778596352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3549 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚3077.138342631 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28070422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7623339 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549332954965504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874104320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7725887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545540063821824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532765184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549332596587008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728753 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33279930 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545540063821824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3787076049715 _step‚293 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14033928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549332954965504 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3792532765184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3833556522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124428464058880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3117.619882043 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124428344286720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596587008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664öOæ'ÎàO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33289907 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3792874104320 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33289907 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120594787764224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14033662 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 119772160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3833657682432 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843658391137571e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33279930 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 trainer/global_step‚3599 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542742991228928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3786735499825 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28444390 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120594806376448 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724318 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549332954965504 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3792874104320 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7725887 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚545540063821824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3792532765184 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚545540080861184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549332596587008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚546529726728753 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33279930 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚545540063821824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3787076049715 _step‚293 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28444388 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14033928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549332954965504 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3792532765184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3833556522496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚124428464058880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚3117.619882043 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚124428344286720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28444390 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549332596587008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚545540080861184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936¥TÊ!*2Ï- _wandbruntime‚3118  _runtime‚3118ÑêDéAÐäM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3834673880576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/reques‹•“©%ted_bytes.large_pool.freed‚542889501900800 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3787746553547 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61750995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚549481208629248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542889518940160 _runtime‚3118.254304549 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28452022 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120628216522752 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33288982 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7727875 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7727598 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚549481208629248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚545687321890816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14037381 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3793886738432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28452020 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61750995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545687304851456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741002 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6309783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3793545392128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3834583213568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546677606043709 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33298973 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14037661 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120628235134976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚546677248454347 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33298973 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124462909015552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28452020 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3599 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚545687321890816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14037381 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3793886738432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28452020 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61750995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚545687304851456 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741002 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6309783 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3793545392128 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3834583213568 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚546677606043709 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33298973 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14037661 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120628235134976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚546677248454347 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33298973 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚124462909015552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¦«âÒÌ2Ò- _wandbruntime‚3118  _runtime‚3118‹M>½Óþ /loss/FRt2/detection_loss‚1.4927392171973082 $loss/total_loss‚9.75078387264815 % _timestamp‚1.7843658397501516e+09 _runtime‚3118.255836802 _step‚295 0loss/FRt1/assignment_loss‚2.1243040378045555 /loss/FRt2/assignment_loss‚1.020517425971073 /loss/FRt1/detection_loss‚0.6057420466287011 0loss/SRqqt1/detection_loss‚1.019260154190568 trainer/global_step‚3599 1loss/FBt1/assignment_loss‚0.01793926894388065 0loss/SRqqt2/detection_loss‚0.620223547205595 /loss/FBt1/detection_loss‚0.5478494901960688 epoch‚8 /loss/FBt2/detection_loss‚1.1234455033471542 2loss/SRqqt1/assignment_loss‚0.6482379094251605 2loss/SRqqt2/assignment_loss‚0.5152635154073004 2loss/FBt2/assignment_loss‚0.015261756330784542ÒXò( Ôƒ 2loss/SRqqt1/assignment_loss‚0.6482379094251605 2loss/SRqqt2/assignment_loss‚0.5152635154073004 2loss/FBt2/assignment_loss‚0.015261756330784542 /loss/FRt2/detection_loss‚1.4927392171973082 $loss/total_loss‚9.75078387264815 % _timestamp‚1.7843658397501516e+09 _runtime‚3118.255836802 _step‚295 0loss/FRt1/assignment_loss‚2.1243040378045555 /loss/FRt2/assignment_loss‚1.020517425971073 /loss/FRt1/detection_loss‚0.6057420466287011 0loss/SRqqt1/detection_loss‚1.019260154190568 trainer/global_step‚3599 1loss/FBt1/assignment_loss‚0.01793926894388065 0loss/SRqqt2/detection_loss‚0.620223547205595 /loss/FBt1/detection_loss‚0.5478494901960688 epoch‚8 /loss/FBt2/detection_loss‚1.1234455033471542§F­Õ‚(:„ ‘†íÒ¬³®ï gpu.0.gpu‚6.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.598 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úKµžªÖ‚(:Ÿ ‘†íÒùò»ñ disk.sda.in‚0) disk./.usagePercent‚39.58846005487818 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0' disk.nvme0c0n1.out‚6809.01611328125 disk.sda1.in‚0 proc.cpu.threads‚17( proc.memory.availableMB‚ 493.82421875$ disk./.usageGB‚589.8966636657715 disk.sda3.out‚68.21875 disk.rbd0.in‚0 disk.nvme0n1.in‚3938.25# proc.memory.rssMB‚ 2771.65234375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚8.3125$ memory_percent‚96.98593616485596* proc.memory.percent‚16.916823387145996 disk.dm-0.in‚3938.25" disk.dm-0.out‚6809.01611328125 disk.nvme0c0n1.in‚3938.25 network.sent‚ 5.665934e+06 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚68.21875 cpu‚25.940474838859892 network.recv‚845983 disk.dm-1.out‚68.21875% disk.nvme0n1.out‚6809.01611328125%ç#“ׂ(:ˆ  †íÒÝä³í gpu.0.gpu‚47.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚255.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê[‘{¶Ø‚(:«  †íÒØ›µï* proc.memory.percent‚16.916823387145996 disk.sda1.in‚0 disk.drbd2165.out‚0 cpu‚25.958040585680674 network.sent‚ 5.728787e+06" disk.nvme0c0n1.in‚ 4085.0390625 disk.sda.out‚ 68.578125 disk.rbd0.in‚0# disk.nvme0n1.out‚6816.373046875 disk.dm-0.out‚6816.373046875 network.recv‚850446 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-0.in‚ 4085.0390625# proc.memory.rssMB‚ 2771.65234375 disk.nvme0n1.in‚ 4085.0390625( proc.memory.availableMB‚ 493.54296875% disk.nvme0c0n1.out‚6816.373046875 disk.sda.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 68.578125 disk.drbd2165.in‚0 disk.rbd0.out‚0$ memory_percent‚96.98765277862549 disk.sda3.out‚ 68.578125 disk.dm-1.in‚0) disk./.usagePercent‚39.58852789687476$ disk./.usageGB‚589.8976745605469 disk.rbd1.out‚8.3125.á¾92Ù- _wandbruntime‚3135  _runtime‚3135¡i("ÚÁ{ ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21289750933647156 )loss/val_total_loss‚9.186018378759929 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898792505264282 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990969181060791 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03756086528301239 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740741789340973 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951962232589722 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6968770027160645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18522685766220093 *jet/accuracy_2_of_4‚0.7864388823509216 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28288495540618896 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02426542341709137 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740741789340973 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 *jet/accuracy_1_of_2‚0.7930887341499329 _step‚296 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6890240907669067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21489311754703522 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5469614863395691 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7938019633293152 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8420820832252502 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1996535360813141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4827914535999298 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7706373333930969 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9973503947257996 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8027289509773254 ,jet/accuracy_4_of_4‚0.009931244887411594 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972979426383972 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16399602591991425 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04361971840262413 +jet/accuracy_2_of_2‚0.15834419429302216 particle/accuracy_2_of_4‚1 % _timestamp‚1.7843658568558137e+09 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6758813858032227 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965227246284485 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04711923003196716 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7473506331443787 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22082878649234772 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21141290664672852 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26408857107162476 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.9761168956756592 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996724009513855 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7678831219673157 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10418523848056793 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039535388350486755 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6889671683311462 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6973167061805725 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.917559027671814 epoch‚8 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6203874349594116 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46346592903137207 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46217772364616394 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6495187282562256 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 'particle/f_score‚0.6330574750900269 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954627752304077 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19369986653327942 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9261786341667175 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7368003726005554 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9929383397102356 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45068395137786865 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1675802320241928 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21224839985370636 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4078335464000702 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6842631697654724 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.765222430229187 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7723835706710815 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03189312666654587 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4968569576740265 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8010016083717346 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚3135.362468306 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.728012204170227 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11974631994962692 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870716333389282 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16678209602832794 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7702005505561829 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3038620352745056 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9811443090438843 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04889397323131561 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7241171598434448 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18005701899528503 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8572067022323608 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7207338809967041 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7063494920730591 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905720949172974 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4189814627170563 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_3_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8937621116638184 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21673354506492615 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8420820832252502 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1996535360813141 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03296752646565437 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945558905601501 9"Purity/0FRt1SRqqt0FBt/wÛ']ŸSRqqt_purity‚0.6880097389221191 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.746414065361023 trainer/global_step‚3599 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7688171863555908 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7813681364059448 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5483506917953491 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 jet/accuracy_2_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963076710700989 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 particle/accuracy_1_of_2‚1 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.039213843643665314 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8698468208312988 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.720086395740509 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08740843087434769 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533333420753479 +jet/accuracy_3_of_3‚0.06864789873361588 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953323602676392 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04393802955746651 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6678619384765625 +particle/specificity‚0.9149447679519653 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7945200800895691 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15318048000335693 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017013033851981163 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19499258697032928 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33398041129112244 *jet/accuracy_2_of_3‚0.5733759999275208 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19243887066841125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2118763029575348 +jet/accuracy_3_of_4‚0.08813785761594772 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7430892586708069 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6107695698738098 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48056742548942566 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9231379628181458 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6367253661155701 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8003301024436951 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939127564430237 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40576377511024475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_4‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9858298301696777 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6839497089385986 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3766149878501892 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16685526072978973 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31958115100860596 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963076710700989 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6880097389221191 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6229906678199768 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7239257097244263 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5862774848937988 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7613781690597534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7666786909103394 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03189312666654587 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19935274124145508 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905350208282471 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10234159231185913 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934483766555786 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6811038255691528 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1794019192457199 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 /particle/accuracy_4_of_4‚0.9505751729011536 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6120200753211975 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7676682472229004 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 (particle/accuracy‚0.8488537073135376 +particle/sensitivity‚0.6071871519088745^ \³Ì=ÛÆ{ 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7473506331443787 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.22082878649234772 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21141290664672852 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26408857107162476 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.9761168956756592 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.996724009513855 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7678831219673157 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10418523848056793 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.039535388350486755 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6889671683311462 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6973167061805725 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.917559027671814 epoch‚8 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6203874349594116 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46346592903137207 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46217772364616394 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6495187282562256 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 'particle/f_score‚0.6330574750900269 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9954627752304077 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19369986653327942 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9261786341667175 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7368003726005554 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583432674408 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9929383397102356 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45068395137786865 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1675802320241928 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21224839985370636 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.4078335464000702 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6842631697654724 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841270446777344 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.765222430229187 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7723835706710815 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03189312666654587 particle/accuracy_1_of_3‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4968569576740265 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8010016083717346 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 _runtime‚3135.362468306 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.728012204170227 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11974631994962692 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9870716333389282 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16678209602832794 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7702005505561829 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3038620352745056 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9811443090438843 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04889397323131561 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7241171598434448 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18005701899528503 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8572067022323608 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7207338809967041 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.7063494920730591 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9905720949172974 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4189814627170563 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_3_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8937621116638184 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21673354506492615 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8420820832252502 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1996535360813141 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03296752646565437 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945558905601501 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6880097389221191 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.746414065361023 trainer/global_step‚3599 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7688171863555908 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7813681364059448 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5483506917953491 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 jet/accuracy_2_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9963076710700989 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 particle/accuracy_1_of_2‚1 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.039213843643665314 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8698468208312988 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.720086395740509 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08740843087434769 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533333420753479 +jet/accuracy_3_of_3‚0.06864789873361588 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9953323602676392 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04393802955746651 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6678619384765625 +particle/specificity‚0.9149447679519653 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7945200800895691 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15318048000335693 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017013033851981163 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19499258697032928 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.33398041129112244 *jet/accuracy_2_of_3‚0.5733759999275208 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19243887066841125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2118763029575348 +jet/accuracy_3_of_4‚0.08813785761594772 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7430892586708069 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6107695698738098 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48056742548942566 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9231379628181458 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6367253661155701 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8003301024436951 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939127564430237 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.40576377511024475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_4‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9858298301696777 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6839497089385986 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3766149878501892 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16685526072978973 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31958115100860596 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9963076710700989 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6880097389221191 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6229906678199768 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7239257097244263 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5862774848937988 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7613781690597534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7666786909103394 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03189312666654587 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.19935274124145508 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.921296238899231 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905350208282471 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10234159231185913 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934483766555786 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6811038255691528 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1794019192457199 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 /particle/accuracy_4_of_4‚0.9505751729011536 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6120200753211975 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7676682472229004 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 (particle/accuracy‚0.8488537073135376 +particle/sensitivity‚0.6071871519088745 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21289750933647156 )loss/val_total_loss‚9.186018378759929 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898792505264282 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990969181060791 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03756086528301239 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.15740741789340973 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951962232589722 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6968770027160645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18522685766220093 *jet/accuracy_2_of_4‚0.7864388823509216 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.28288495540618896 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02426542341709137 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.15740741789340973 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9922618865966797 *jet/accuracy_1_of_2‚0.7930887341499329 _step‚296 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6890240907669067 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21489311754703522 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5469614863395691 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7938019633293152 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8420820832252502 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1996535360813141 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4827914535999298 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7706373333930969 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9973503947257996 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8027289509773254 ,jet/accuracy_4_of_4‚0.009931244887411594 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972979426383972 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16399602591991425 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04361971840262413 +jet/accuracy_2_of_2‚0.15834419429302216 particle/accuracy_2_of_4‚1 % _timestamp‚1.7843658568558137e+09 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6758813858032227 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9965227246284485 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04711923003196716 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625¨…?J[žÜ‚(:“ ¯†íÒáƒÂî gpu.0.gpu‚98.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚253.755' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.751000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Õìö·Ý‚(:¬ ¯†íÒá»Ðï disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4085.0390625& disk.nvme0c0n1.out‚7066.0869140625 disk.sda1.out‚0$ disk./.usageGB‚589.8985176086426 disk.drbd2165.in‚0 disk.sda.in‚0$ memory_percent‚95.45919895172119$ disk.nvme0n1.out‚7066.0869140625 disk.sda1.in‚0* proc.memory.percent‚16.916823387145996 proc.cpu.threads‚17 disk.sda3.out‚ 68.796875 disk.rbd0.in‚0! disk.dm-0.out‚7066.0869140625 disk.sda3.in‚0 disk.dm-1.out‚ 68.796875 disk.drbd2165.out‚0 disk.dm-0.in‚ 4085.0390625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚8.375( proc.memory.availableMB‚ 743.96484375 network.sent‚ 5.733691e+06 disk.dm-1.in‚0# proc.memory.rssMB‚ 2771.65234375 network.recv‚853931) disk./.usagePercent‚39.58858447453982 disk.nvme0n1.in‚ 4085.0390625 disk.sda2.in‚0 disk.sda.out‚ 68.796875 cpu‚25.95327426948033y£Û¿“Þ‚(:ˆ ¾†íÒã÷‘ï gpu.0.gpu‚73.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l„ù„±ß‚(:¦ ¾†íÒô™ñ disk.rbd0.out‚0 disk.nvme0n1.in‚ 4085.0390625 disk.sda.in‚0 disk.sda.out‚ 69.08984375 proc.cpu.threads‚17 network.recv‚858025 disk.sda1.in‚0! disk.nvme0n1.out‚ 7317.2109375 disk.sda2.out‚0 disk.rbd1.out‚8.375 disk.sda3.out‚ 69.08984375 disk.rbd0.in‚0 disk.dm-0.out‚ 7317.2109375* proc.memory.percent‚16.916823387145996 network.sent‚ 5.793169e+06$ disk./.usageGB‚589.8975296020508 disk.sda3.in‚0 disk.dm-1.out‚ 69.08984375 disk.drbd2165.out‚0# memory_percent‚95.4568862915039% proc.memory.availableMB‚ 744.34375 disk.dm-1.in‚0 disk.dm-0.in‚ 4085.0390625# disk.nvme0c0n1.out‚ 7317.2109375) disk./.usagePercent‚39.58851816858846 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 4085.0390625# proc.memory.rssMB‚ 2771.65234375 cpu‚25.952056683070474 disk.sda1.out‚0JzÅÛ2à- _wandbruntime‚3178  _runtime‚3178ýV,ÿÇáÁ % _timestamp‚1.7843658996660795e+09 _runtime‚3178.172068938 _step‚297 & lr-AdamW/pg1‚0.0005434456015727306 & lr-AdamW/pg2‚0.0005434456015727306 trainer/global_step‚3649¼Q­ÌâÆ _step‚297 & lr-AdamW/pg1‚0.0005434456015727306 & lr-AdamW/pg2‚0.0005434456015727306 trainer/global_step‚3649 % _timestamp‚1.7843658996660795e+09 _runtime‚3178.172068938©> \‘2ã- _wandbruntime‚3178  _runtime‚3178ÕU”¸äÚO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751363 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554831698993152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚3178.173255598 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643428864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33804709 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3902933669376 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pÒ¼rc(ool.allocated‚ 3903018052096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3861302814208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643428864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33814686 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3860961475072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14258278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3649 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555832617337825 _step‚298 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554831681953792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7860738 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261857 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33804709 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814686 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3861302814208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14258549 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558693001807360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3860961475072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269Ñ×Oå'åßO KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28946656 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚551977188036608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28946654 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6397808 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3903018052096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3861302814208 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28946656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558692643428864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33814686 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3860961475072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14258278 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 trainer/global_step‚3649 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚555832617337825 _step‚298 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚554831681953792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7860738 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3855412261857 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33804709 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33814686 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3861302814208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14258549 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558693001807360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3860961475072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751363 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚554831698993152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚3178.173255598 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558692643428864 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33804709 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3902933669376 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554978922983424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122315585161216 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62768047 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14262048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555980139063531 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7862495 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784365900305853e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861974102016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126219535112192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555979781474169 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554978940022784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255471104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862315448320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33823752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7862766 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚552123698708480 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315448320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861974102016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6399553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33823752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33813761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28954286 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚558840897085440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3904046647296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚3178.812062465 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3856423315691 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28954288«Û9ƒ¶"èçM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28954286 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚3649 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚558840897085440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3904046647296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚78084096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚3178.812062465 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3856423315691 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28954288 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚299 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554978922983424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚122315585161216 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62768047 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14262048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚555980139063531 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7862495 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 $ _timestamp‚1.784365900305853e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3861974102016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126219535112192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚555979781474169 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚552123715747840 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554978940022784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558841255471104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3862315448320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33823752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7862766 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚552123698708480 OADeviceStatsMonitor.on_train_batch_end/allocated_ÒÃIÞ7bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3862315448320 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3861974102016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6399553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33823752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33813761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438«G8zõ2é- _wandbruntime‚3178  _runtime‚3178v¤E’ê‚ /loss/FRt2/detection_loss‚1.4513240030655294 trainer/global_step‚3649 % _timestamp‚1.7843659003076503e+09 /loss/FBt2/detection_loss‚0.9076518000103533 _runtime‚3178.813337081 _step‚300 /loss/FRt1/assignment_loss‚2.118625179070131 1loss/SRqqt1/detection_loss‚1.0986434481037426 /loss/FBt1/detection_loss‚0.4037275097687684 0loss/FRt2/assignment_loss‚0.9699994515681613 .loss/FRt1/detection_loss‚0.609079284398153 1loss/SRqqt2/detection_loss‚0.7045629801360409 %loss/total_loss‚9.516097775167022 epoch‚9 2loss/SRqqt1/assignment_loss‚0.7397717487996383 2loss/SRqqt2/assignment_loss‚0.4924796617643141 2loss/FBt1/assignment_loss‚0.017128943216899388 3loss/FBt2/assignment_loss‚0.0031037652652898375î Î ë‡ epoch‚9 2loss/SRqqt1/assignment_loss‚0.7397717487996383 2loss/SRqqt2/assignment_loss‚0.4924796617643141 2loss/FBt1/assignment_loss‚0.017128943216899388 3loss/FBt2/assignment_loss‚0.0031037652652898375 /loss/FRt2/detection_loss‚1.4513240030655294 trainer/global_step‚3649 % _timestamp‚1.7843659003076503e+09 /loss/FBt2/detection_loss‚0.9076518000103533 _runtime‚3178.813337081 _step‚300 /loss/FRt1/assignment_loss‚2.118625179070131 1loss/SRqqt1/detection_loss‚1.0986434481037426 /loss/FBt1/detection_loss‚0.4037275097687684 0loss/FRt2/assignment_loss‚0.9699994515681613 .loss/FRt1/detection_loss‚0.609079284398153 1loss/SRqqt2/detection_loss‚0.7045629801360409 %loss/total_loss‚9.516097775167022¬AtÕŸ‘ì‚(:† ͆íÒœ™Ëí gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚239.197' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.8394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ƒÜÀí‚(:µ ͆íÒÍ‘Øî# proc.memory.rssMB‚ 2771.65234375 disk.rbd0.in‚0% disk.nvme0n1.out‚7320.36767578125 cpu‚25.950919274400178 disk.nvme0n1.in‚ 4085.0390625" disk.nvme0c0n1.in‚ 4085.0390625 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚8.5$ memory_percent‚95.45981884002686 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 69.45703125 disk.drbd2165.out‚0 disk.rbd0.out‚0* proc.memory.percent‚16.916823387145996 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 69.45703125 disk.sda1.in‚0 disk.dm-0.in‚ 4085.0390625 disk.sda2.in‚0( proc.memory.availableMB‚ 743.86328125 network.recv‚861648$ disk./.usageGB‚589.8975219726562 disk.drbd2165.in‚0" disk.dm-0.out‚7320.36767578125 network.sent‚ 5.796192e+06* disk./.usagePercent‚39.588517656573394' disk.nvme0c0n1.out‚7320.36767578125 disk.dm-1.out‚ 69.45703125=4†î‚(:’ ܆íÒÌ‘üì gpu.0.gpu‚87.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚236.479' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.29580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?™mÛ¼ï‚(:± ܆íÒµ…Äï disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚7327.44677734375" disk.dm-0.out‚7327.44677734375 disk.rbd1.out‚8.5* disk./.usagePercent‚39.588513560452846 disk.dm-1.out‚ 69.83984375 disk.dm-0.in‚ 4085.0390625' disk.nvme0c0n1.out‚7327.44677734375* proc.memory.percent‚16.916823387145996" disk.nvme0c0n1.in‚ 4085.0390625 disk.sda1.out‚0( proc.memory.availableMB‚ 743.88671875# proc.memory.rssMB‚ 2771.65234375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 4085.0390625 disk.sda3.out‚ 69.83984375# memory_percent‚95.4596757888794! disk./.usageGB‚589.8974609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 69.83984375 disk.sda2.in‚0 disk.sda.in‚0 cpu‚25.949658607411497 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.out‚0 network.sent‚ 5.859369e+06 network.recv‚866306ÅC9Ú‘ð‚(:† ë†íÒîÅÜî gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŋ趴ñ‚(:© ë†íҶſð network.recv‚870029* disk./.usagePercent‚39.588513048437775& proc.memory.availableMB‚ 744.484375 disk.dm-0.in‚ 4192.796875 disk.sda.out‚70.15625 disk.sda1.in‚0$ memory_percent‚95.45602798461914* proc.memory.percent‚16.916823387145996% disk.nvme0n1.out‚7330.59228515625 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚8.5625 disk.rbd0.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚7330.59228515625 disk.sda1.out‚0 disk.sda3.out‚70.15625 disk.dm-1.out‚70.15625 disk.drbd2165.in‚0$ disk./.usageGB‚589.8974533081055" disk.dm-0.out‚7330.59228515625# proc.memory.rssMB‚ 2771.65234375 cpu‚25.94856193317287 proc.cpu.threads‚17 network.sent‚ 5.866239e+06 disk.nvme0n1.in‚ 4192.796875! disk.nvme0c0n1.in‚ 4192.796875ÙD‚2ò- _wandbruntime‚3219  _runtime‚32195>ãñÇóÁ & lr-AdamW/pg1‚0.0005422771100532271 & lr-AdamW/pg2‚0.0005422771100532271 trainer/global_step‚3699 % _timestamp‚1.7843659405039735e+09 _runtime‚3219.009935066 _step‚301E¾âÖÌôÆ _step‚301 & lr-AdamW/pg1‚0.0005422771100532271 & lr-AdamW/pg2‚0.0005422771100532271 trainer/global_step‚3699 % _timestamp‚1.7843659405039735e+09 _runtime‚3219.009935066­"…è2õ- _wandbruntime‚3219  _runtime‚3219hooUÞ'öØO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911656976896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3911998316032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14449434 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3905688198875 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34259161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6485061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6485058 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562193733435392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911998316032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚563208409829083 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597394 _step‚302 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390412288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34259161 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123953594329088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587415 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127907859569152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚3219.011132236 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34269138 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566105390412288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748790784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29328256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34269138 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29328254 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14449694 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562193750474752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562193750474752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127907956272640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3906028748765 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784365940504999e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3954361943552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976Ë5ã'÷ÝO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29328256 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3911656976896 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3911998316032 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14449434 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3905688198875 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34259161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6485061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6485058 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562193733435392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3911998316032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚563208409829083 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚559302738669568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597394 _step‚302 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566105390412288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34259161 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123953594329088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587415 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127907859569152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _runtime‚3219.011132236 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34269138 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚96703488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚566105390412288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚566105748790784 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29328256 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34269138 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29328254 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14449694 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562193750474752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562193750474752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127907956272640 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3906028748765 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 $ _timestamp‚1.784365940504999e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3954361943552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0® q¤k2ø- _wandbruntime‚3219  _runtime‚32193Ñj+ùãM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34268213 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63614092 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604099 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚559449249341440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚566253644068864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6486806 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚3699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚562340974465024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127942413505536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278204 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14453492 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843659411344676e+09 _runtime‚3219.640695803 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335886 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚303 _KDeviceStats' µV¾Monitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123987004475392 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566254002454528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3906699252597 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566254002454528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562340974465024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127942314712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3955390417920 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614092 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123987023087616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6486803 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29335888 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604099 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34278204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3913010950144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚563356289144039 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7966433 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3955310236672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3913010950144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14453236 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34278204 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3913010950144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚563356289144039 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7966433 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3955310236672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3913010950144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14453236 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34278204 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14453492 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843659411344676e+09 _runtime‚3219.640695803 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29335886 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚123987004475392 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚566254002454528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3906699252597 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566254002454528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚562340974465024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚127942314712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3955390417920 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614092 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29335888 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123987023087616 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6486803 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304¯Ô=!2û- _wandbruntime‚3219  _runtime‚3219d•\Ùüÿ 1loss/SRqqt2/assignment_loss‚0.609068184536808 /loss/FRt1/detection_loss‚0.6323908363052211 /loss/FBt1/detection_loss‚0.6262300212130261 /loss/FBt2/detection_loss‚0.7943653451169238 _runtime‚3219.641609979 0loss/FRt1/assignment_loss‚2.0175847353372482 1loss/FBt2/assignment_loss‚0.09492336535905611 /loss/FRt2/detection_loss‚1.4569833574140059 %loss/total_loss‚9.524233519821532 % _timestamp‚1.7843659411356516e+09 _step‚304 1loss/FBt1/assignment_loss‚0.05836655962811624 0loss/SRqqt2/detection_loss‚0.666638063919404 1loss/SRqqt1/detection_loss‚0.9498166510169088 trainer/global_step‚3699 epoch‚9 0loss/FRt2/assignment_loss‚0.9662367320285653 2loss/SRqqt1/assignment_loss‚0.6516296679462497Û*hL ý„ 2loss/SRqqt1/assignment_loss‚0.6516296679462497 1loss/SRqqt2/assignment_loss‚0.609068184536808 /loss/FRt1/detection_loss‚0.6323908363052211 /loss/FBt1/detection_loss‚0.6262300212130261 /loss/FBt2/detection_loss‚0.7943653451169238 _runtime‚3219.641609979 0loss/FRt1/assignment_loss‚2.0175847353372482 1loss/FBt2/assignment_loss‚0.09492336535905611 /loss/FRt2/detection_loss‚1.4569833574140059 %loss/total_loss‚9.524233519821532 % _timestamp‚1.7843659411356516e+09 _step‚304 1loss/FBt1/assignment_loss‚0.05836655962811624 0loss/SRqqt2/detection_loss‚0.666638063919404 1loss/SRqqt1/detection_loss‚0.9498166510169088 trainer/global_step‚3699 epoch‚9 0loss/FRt2/assignment_loss‚0.9662367320285653°šø‰ãþ‚(:’ ú†íÒ£§üí gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïú¯ÿ‚(:¤ ú†íÒˆµ¿ï disk.sda2.out‚0 disk.dm-1.out‚70.4375 disk.drbd2165.in‚0$ memory_percent‚95.45950889587402* proc.memory.percent‚16.916823387145996$ disk.nvme0n1.out‚7335.4599609375 disk.dm-0.in‚ 4192.796875& disk.nvme0c0n1.out‚7335.4599609375 disk.sda.in‚0 disk.sda1.in‚0) disk./.usagePercent‚39.58849256783503 disk.rbd1.out‚8.5625 disk.sda3.in‚0' proc.memory.availableMB‚ 743.9140625 cpu‚25.947431286988714 network.sent‚ 5.869161e+06 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2771.65234375 disk.nvme0n1.in‚ 4192.796875 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 4192.796875 disk.sda.out‚70.4375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.out‚70.4375 proc.cpu.threads‚17 network.recv‚873475$ disk./.usageGB‚589.8971481323242! disk.dm-0.out‚7335.4599609375 disk.sda2.in‚05œž€‚(:“ ‰‡íÒîÂîí gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚245.379' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.075799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/᡺‚(:¯ ‰‡íÒÕ´Ñî* disk./.usagePercent‚39.588530200942564$ disk./.usageGB‚589.8977088928223 disk.rbd1.in‚0 disk.rbd1.out‚8.625 disk.sda3.out‚ 70.8359375 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚95.45760154724121 network.sent‚ 5.932233e+06% disk.nvme0n1.out‚7338.21435546875 disk.sda.out‚ 70.8359375 disk.sda1.in‚0' proc.memory.availableMB‚ 744.2265625 disk.dm-1.out‚ 70.8359375 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 4192.796875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 4192.796875! disk.nvme0c0n1.in‚ 4192.796875 disk.sda3.in‚0 network.recv‚878191 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.916823387145996" disk.dm-0.out‚7338.21435546875' disk.nvme0c0n1.out‚7338.21435546875 disk.sda1.out‚0# proc.memory.rssMB‚ 2771.65234375 cpu‚25.94615563735589É'€r“‚‚(:ˆ ˜‡íÒﵸï gpu.0.gpu‚96.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚261.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÚ®«ƒ‚(:  ˜‡íÒܹáñ disk.sda3.out‚71.0625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 744.0546875* proc.memory.percent‚16.916823387145996 network.sent‚ 5.939296e+06 disk.rbd1.out‚8.625! disk.nvme0c0n1.in‚ 4192.796875 disk.sda.out‚71.0625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 4192.796875 disk.dm-0.in‚ 4192.796875 disk.sda1.in‚0 network.recv‚881785 disk.dm-1.out‚71.0625 disk.rbd0.in‚0 disk.dm-0.out‚7344.126953125$ disk./.usageGB‚589.8979110717773 disk.rbd1.in‚0% disk.nvme0c0n1.out‚7344.126953125 disk.dm-1.in‚0 cpu‚25.944902099527543 disk.sda.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2771.65234375) disk./.usagePercent‚39.58854376934188# disk.nvme0n1.out‚7344.126953125$ memory_percent‚95.45865058898926 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0Ó}2„- _wandbruntime‚3260  _runtime‚3260/n¿ŸÆ…À & lr-AdamW/pg2‚0.0005410924802038668 trainer/global_step‚3749 $ _timestamp‚1.784365981529145e+09 _runtime‚3260.035121247 _step‚305 & lr-AdamW/pg1‚0.0005410924802038668§MˆŠ_runtime‚3260.035121247 _step‚305 & lr-AdamW/pg1‚0.0005410924802038668 & lr-AdamW/pg2‚0.0005410924802038668 trainer/global_step‚3749 $ _timestamp‚1.784365981529145e+09±§mƒ72‡- _wandbruntime‚3260  _runtime‚3260µB œà'ˆÚO trainer/global_step‚3749 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34723590 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573518137395712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚569555784916992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64423467 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3260.036423075 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29709856 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137395712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14640652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14640381 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3956645235673 % _timestamp‚1.7843659815302885e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6572308 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8068341 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125625032255488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚569555801956352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573518495774208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34723590 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29709854 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8068073 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129630485521920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚570584917498841 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚306 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚570584559909591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566628255223808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573518137395712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14640652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14640381 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3956645235673 % _timestamp‚1.7843659815302885e+09 A7DevicÖçYÀ#eStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6572308 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8068341 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125625032255488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚569555801956352 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚573518495774208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34723590 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433446 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29709854 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8068073 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129630485521920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚570584917498841 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 _step‚306 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚570584559909591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚566628255223808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976²%™ÏI2Š- _wandbruntime‚3260  _runtime‚3260Rù>é&‹ãM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34722665 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4006482563072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957315739505 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29717488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569703042985984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6574053 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573666391052288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3957656289507 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8070165 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129665040535552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566774765895680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843659821625993e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34732656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963365105664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963706451968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14644218 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚307 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8070441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29717486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440151 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29717486 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569703025946624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34722665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976‡kIßî&ŒèM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3957656289507 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8070165 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129665040535552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚566774765895680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 % _timestamp‚1.7843659821625993e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34732656 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3963365105664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3963706451968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14644218 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚307 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8070441 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29717486 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440151 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29717486 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34732656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚569703025946624 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34722665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34722665 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4006482563072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3957315739505 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29717488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚569703042985984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6574053 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573666391052288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0³ÞX¥S2- _wandbruntime‚3260  _runtime‚3260×ÈçX Žƒ %loss/total_loss‚9.594635111129005 epoch‚9 0loss/FRt1/assignment_loss‚2.0251020219313536 3loss/FBt1/assignment_loss‚0.0065825911547295534 /loss/FRt1/detection_loss‚0.6508795049489038 1loss/SRqqt1/detection_loss‚0.9046350778719056 1loss/SRqqt2/detection_loss‚0.7711677861054773 _step‚308 0loss/FRt2/assignment_loss‚1.0267735409784127 2loss/SRqqt1/assignment_loss‚0.6239076283975289 /loss/FRt2/detection_loss‚1.4511879944275932 /loss/FBt1/detection_loss‚0.4171128143852418 trainer/global_step‚3749 _runtime‚3260.670008607 2loss/SRqqt2/assignment_loss‚0.5371307292888872 .loss/FBt2/detection_loss‚1.169501482276246 % _timestamp‚1.7843659821643267e+09 2loss/FBt2/assignment_loss‚0.010653939362725167øY›Ôˆ 2loss/SRqqt2/assignment_loss‚0.5371307292888872 .loss/FBt2/detection_loss‚1.169501482276246 % _timestamp‚1.7843659821643267e+09 2loss/FBt2/assignment_loss‚0.010653939362725167 %loss/total_loss‚9.594635111129005 epoch‚9 0loss/FRt1/assignment_loss‚2.0251020219313536 3loss/FBt1/assignment_loss‚0.0065825911547295534 /loss/FRt1/detection_loss‚0.6508795049489038 1loss/SRqqt1/detection_loss‚0.9046350778719056 1loss/SRqqt2/detection_loss‚0.7711677861054773 _step‚308 0loss/FRt2/assignment_loss‚1.0267735409784127 2loss/SRqqt1/assignment_loss‚0.6239076283975289 /loss/FRt2/detection_loss‚1.4511879944275932 /loss/FBt1/detection_loss‚0.4171128143852418 trainer/global_step‚3749 _runtime‚3260.670008607´/9r%ž‚(:“ §‡íÒüÿ©ï gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ¹‘‚(:® §‡íÒŒªÏñ$ memory_percent‚95.45745849609375 disk.sda3.out‚ 71.3203125' disk.nvme0c0n1.out‚7347.78271484375 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd1.in‚0 network.recv‚885418 disk.sda2.in‚0" proc.memory.availableMB‚744.25* proc.memory.percent‚16.916823387145996 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚8.6875$ disk./.usageGB‚589.8978958129883 disk.sda.out‚ 71.3203125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 4269.2734375 disk.dm-0.in‚ 4269.2734375 cpu‚25.943941232667683 network.sent‚ 5.942319e+06) disk./.usagePercent‚39.58854274531174 disk.dm-1.out‚ 71.3203125 disk.rbd0.out‚0" disk.dm-0.out‚7347.78271484375 disk.sda1.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚7347.78271484375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 4269.2734375# proc.memory.rssMB‚ 2771.65234375¡GŒ’‚(:’ ¶‡íÒá¢êì gpu.0.gpu‚76.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚236.479' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.29580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èkm“‚(:· ¶‡íÒöÇ’ð disk.rbd1.out‚8.6875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 4269.2734375( proc.memory.availableMB‚ 744.30859375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2771.65234375' disk.nvme0c0n1.out‚7353.58251953125 disk.sda1.in‚0 disk.sda1.out‚0" disk.dm-0.out‚7353.5825195Qœ~U3125 network.sent‚ 6.007198e+06 disk.sda.in‚0 disk.drbd2165.out‚0# memory_percent‚95.4571008682251 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 71.65234375 disk.dm-0.in‚ 4269.2734375 disk.sda3.out‚ 71.65234375 disk.dm-1.in‚0 disk.dm-1.out‚ 71.65234375 cpu‚25.942708729740804 network.recv‚889462* disk./.usagePercent‚39.588530456950096" disk.nvme0c0n1.in‚ 4269.2734375 disk.drbd2165.in‚0* proc.memory.percent‚16.916823387145996$ disk./.usageGB‚589.8977127075195 disk.sda3.in‚0% disk.nvme0n1.out‚7353.58251953125š9“”‚(:ˆ ŇíÒý ùì gpu.0.gpu‚94.0 gpu.0.memory‚32+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬äpº•‚(:¯ ŇíÒÖüð" disk.dm-0.out‚7357.04931640625 disk.sda2.in‚0' disk.nvme0c0n1.out‚7357.04931640625 disk.dm-1.out‚ 71.953125 disk.sda.out‚ 71.953125 disk.drbd2165.in‚0$ memory_percent‚95.46034336090088 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚8.75 disk.sda1.in‚0# proc.memory.rssMB‚ 2771.65234375 disk.sda.in‚0( proc.memory.availableMB‚ 743.77734375 network.recv‚893064) disk./.usagePercent‚39.58849512791038$ disk./.usageGB‚589.8971862792969 disk.sda3.in‚0 disk.rbd1.in‚0 cpu‚25.941600221537872 disk.nvme0n1.in‚ 4269.2734375 disk.sda1.out‚0% disk.nvme0n1.out‚7357.04931640625 network.sent‚ 6.010162e+06 disk.sda3.out‚ 71.953125 disk.rbd0.in‚0 disk.dm-0.in‚ 4269.2734375* proc.memory.percent‚16.916823387145996 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 4269.2734375~2é–2–- _wandbruntime‚3300  _runtime‚3300\Án¡Ç—Á & lr-AdamW/pg1‚0.0005398917881112109 & lr-AdamW/pg2‚0.0005398917881112109 trainer/global_step‚3799 % _timestamp‚1.7843660223760834e+09 _runtime‚3300.882073422 _step‚309Þ¶5Ì˜Æ & lr-AdamW/pg1‚0.0005398917881112109 & lr-AdamW/pg2‚0.0005398917881112109 trainer/global_step‚3799 % _timestamp‚1.7843660223760834e+09 _runtime‚3300.882073422 _step‚309µ›žd-2™- _wandbruntime‚3300  _runtime‚3300x£Úà'šÚO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4013047980544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013389319680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127296451569664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3799 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884379136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577961067579349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259519 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14831759 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚3300.885763753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242757632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389319680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269498 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6659561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8172198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131353097485824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30091454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884379136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580931242757632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168065 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4056542921216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573953788817408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576917853437952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168065 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14831512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30091454 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4013047980544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172691 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30091456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169„ã¤å'›ßO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35178042 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8171954 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚580930884379136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577961067579349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65259519 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14831759 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _runtime‚3300.885763753 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚580931242757632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4013389319680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚65269498 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6659561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8172198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131353097485824 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚127296470181888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30091454 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚580930884379136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚580931242757632 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35168065 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚576917836398592 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚576917836398592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4056542921216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573953788817408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚576917853437952 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35168065 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14831512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30091454 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4013047980544 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573953805856768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4006921172691 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30091456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚577960709990099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4013047980544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013389319680 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚127296451569664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3799 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4014401953792 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 $ _timestamp‚1.784366023007426e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401953792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131387444849664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30099086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14835336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127329898940416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4014060607488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286196 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276203 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35177117 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14835610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6661303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138035712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578108589305055 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚311 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581079496421376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚574100316528640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3301.513547331 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269g†T$î&žèM _runtime‚3301.513547331 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35187108 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8174304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚3799 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4014060607488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚577065094467584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4014401953792 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 $ _timestamp‚1.784366023007426e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014401953792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131387444849664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30099086 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14835336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚127329898940416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4014060607488 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286196 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65276203 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35177117 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14835610 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚127329880328192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6661303 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚581079138035712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚578108589305055 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _step‚311 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚581079496421376 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚574100316528640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0·£{6ª2Ÿ- _wandbruntime‚3301  _runtime‚3301ñW û 1loss/FBt2/assignment_loss‚0.01455095537674627 /loss/FRt1/detection_loss‚0.5921652791274984 1loss/SRqqt1/detection_loss‚1.0090823258267032 $loss/total_loss‚9.74736040104901 trainer/global_step‚3799 $ _timestamp‚1.784366023008986e+09 0loss/FRt2/assignment_loss‚1.1244752200331698 1loss/SRqqt2/assignment_loss‚0.590172795256394 1loss/FBt1/assignment_loss‚0.08076285479474617 .loss/FRt2/detection_loss‚1.409602476109639 0loss/SRqqt2/detection_loss‚0.672746967759941 /loss/FBt2/detection_loss‚0.8829456810273376 epoch‚9 _step‚312 0loss/FRt1/assignment_loss‚2.0904807130700536 2loss/SRqqt1/assignment_loss‚0.7991651992236346 .loss/FBt1/detection_loss‚0.481209933443146 _runtime‚3301.514679149ÿ?Dú¡€ epoch‚9 _step‚312 0loss/FRt1/assignment_loss‚2.0904807130700536 2loss/SRqqt1/assignment_loss‚0.7991651992236346 .loss/FBt1/detection_loss‚0.481209933443146 _runtime‚3301.514679149 1loss/FBt2/assignment_loss‚0.01455095537674627 /loss/FRt1/detection_loss‚0.5921652791274984 1loss/SRqqt1/detection_loss‚1.0090823258267032 $loss/total_loss‚9.74736040104901 trainer/global_step‚3799 $ _timestamp‚1.784366023008986e+09 0loss/FRt2/assignment_loss‚1.1244752200331698 1loss/SRqqt2/assignment_loss‚0.590172795256394 1loss/FBt1/assignment_loss‚0.08076285479474617 .loss/FRt2/detection_loss‚1.409602476109639 0loss/SRqqt2/detection_loss‚0.672746967759941 /loss/FBt2/detection_loss‚0.8829456810273376¸‹™r»“¢‚(:ˆ Ô‡íÒÝø÷ï gpu.0.gpu‚63.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚177.504' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f&e À£‚(:µ Ô‡íÒ‹¿¸ò disk.nvme0n1.in‚ 4269.2734375" disk.dm-0.out‚7362.60009765625 disk.sda.in‚0 disk.sda3.in‚0 cpu‚25.940565960260614 disk.dm-0.in‚ 4269.2734375 disk.rbd1.out‚8.75 disk.sda1.out‚0 disk.sda3.out‚ 72.28515625 disk.sda2.in‚0' disk.nvme0c0n1.out‚7362.60009765625% disk.nvme0n1.out‚7362.60009765625 network.sent‚ 6.028143e+06 network.recv‚896556# proc.memory.rssMB‚ 2771.65234375* proc.memory.percent‚16.916823387145996 proc.cpu.threads‚17$ disk./.usageGB‚589.8972244262695 disk.drbd2165.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 743.93359375 disk.dm-1.in‚0 disk.dm-1.out‚ 72.28515625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚95.45938968658447) disk./.usagePercent‚39.58849768798572 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4269.2734375 disk.sda.out‚ 72.28515625¹8‚’¤‚(:‡ ã‡íÒ¾üí gpu.0.gpu‚13.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sHú¼¥‚(:± ã‡íÒÏô³ï disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 4355.39453125) disk./.usagePercent‚39.58850588022681 disk.sda2.out‚0 disk.sda.out‚ 72.6953125 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚ 6.078083e+06 network.recv‚900949 disk.rbd0.in‚0$ disk.nvme0n1.out‚7366.5517578125 disk.sda2.in‚0 disk.rbd1.out‚8.875 disk.dm-0.in‚ 4355.39453125 disk.sda1.out‚0 disk.sda3.out‚ 72.6953125 disk.dm-1.out‚ 72.6953125& disk.nvme0c0n1.out‚7366.5517578125( proc.memory.availableMB‚ 744.17578125 disk.rbd0.out‚0! disk.nvme0n1.in‚ 4355.39453125* proc.memory.percent‚16.916823387145996# disk./.usageGB‚589.897346496582! disk.dm-0.out‚7366.5517578125 disk.sda.in‚0$ memory_percent‚95.45791149139404# proc.memory.rssMB‚ 2771.65234375 disk.sda1.in‚0 cpu‚25.93939041211905 proc.cpu.threads‚17ך­€2¦- _wandbruntime‚3341  _runtime‚33416É¿ÖǧÁ & lr-AdamW/pg1‚0.0005386751108934688 & lr-AdamW/pg2‚0.0005386751108934688 trainer/global_step‚3849 % _timestamp‚1.7843660634525461e+09 _runtime‚3341.958551413 _step‚313ç¸cÌ¨Æ & lr-AdamW/pg1‚0.0005386751108934688 & lr-AdamW/pg2‚0.0005386751108934688 trainer/global_step‚3849 % _timestamp‚1.7843660634525461e+09 _runtime‚3341.958551413 _step‚313¹26 ;2©- _wandbruntime‚3341  _runtime‚3341}³¤ôà'ªÚO IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843660634537573e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584279904919552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15022020 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4063743482368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6746808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚581279339450368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095571 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128967908108288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4107742812672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4057878209489 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584279904919552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚581279322411008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15022302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6746811 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8275491 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588343631362560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631362560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584279887880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4064084821504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843660634537573e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚584279904919552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15022020 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4063743482368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6746808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚581279339450368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095571 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128967908108288 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4107742812672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35622517 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4057878209489 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚584279904919552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚581279322411008 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15022302 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6746811 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8275491 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚588343631362560 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚588343631362560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚584279887880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4064084821504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15025818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588491885019136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112255 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6748553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133109979111424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584427145949184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30480688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243404800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8277530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129001336866816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585484381796201 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4058889263323 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581425833082880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35631569 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚3342.589743495 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6748556 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30480688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15026086 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚315 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133110096779264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8277265 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4108759912448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843660640835545e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4108660856832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30480686 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129001318254592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097455616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756109312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 trainer/global_step‚3849 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983hwJï&®éM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30480686 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6748556 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35641560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30480688 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15026086 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚315 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133110096779264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8277265 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4108759912448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843660640835545e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4108660856832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30480686 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129001318254592 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4065097455616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4064756109312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 trainer/global_step‚3849 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15025818 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚588491885019136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112255 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6748553 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133109979111424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚584427145949184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30480688 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚588492243404800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8277530 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129001336866816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚585484381796201 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4058889263323 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581425833082880 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35631569 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 _runtime‚3342.589743495»ªÄ²˜2¯- _wandbruntime‚3342  _runtime‚3342—¼°ÿ /loss/FBt2/detection_loss‚0.7144844149658968 % _timestamp‚1.7843660640852098e+09 _step‚316 1loss/SRqqt1/assignment_loss‚0.650680375026832 1loss/SRqqt1/detection_loss‚0.9982936010419784 /loss/FBt1/detection_loss‚0.4429938457906246 trainer/global_step‚3849 2loss/SRqqt2/assignment_loss‚0.5091906973135548 1loss/SRqqt2/detection_loss‚0.6323645787731725 %loss/total_loss‚8.918360315733063 1loss/FBt2/assignment_loss‚0.01376925849557622 /loss/FRt2/detection_loss‚1.4309777482660448 epoch‚9 _runtime‚3342.590889585 .loss/FRt1/assignment_loss‚1.96503807075562 0loss/FRt2/assignment_loss‚0.9517056676820352 2loss/FBt1/assignment_loss‚0.004978874877965169 /loss/FRt1/detection_loss‚0.6038831827437617Ï èE ±„ /loss/FBt2/detection_loss‚0.7144844149658968 % _timestamp‚1.7843660640852098e+09 _step‚316 1loss/SRqqt1/assignment_loss‚0.650680375026832 1loss/SRqqt1/detection_loss‚0.9982936010419784 /loss/FBt1/detection_loss‚0.4429938457906246 trainer/global_step‚3849 2loss/SRqqt2/assignment_loss‚0.5091906973135548 1loss/SRqqt2/detection_loss‚0.6323645787731725 %loss/total_loss‚8.918360315733063 1loss/FBt2/assignment_loss‚0.01376925849557622 /loss/FRt2/detection_loss‚1.4309777482660448 epoch‚9 _runtime‚3342.590889585 .loss/FRt1/assignment_loss‚1.96503807075562 0loss/FRt2/assignment_loss‚0.9517056676820352 2loss/FBt1/assignment_loss‚0.004978874877965169 /loss/FRt1/detection_loss‚0.6038831827437617¼‹x:)‘²‚(:† ò‡íÒ¢Èèï gpu.0.gpu‚86.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚223.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lé,œÄ³‚(:¹ ò‡íÒ¿¦˜ò disk.rbd1.out‚8.875' disk.nvme0c0n1.out‚7373.95458984375 disk.sda.out‚ 72.94921875 network.sent‚ 6.082946e+06 disk.sda3.out‚ 72.94921875 disk.rbd0.out‚0 disk.dm-0.in‚ 4355.39453125$ disk./.usageGB‚589.8973274230957 disk.rbd0.in‚0" disk.dm-0.out‚7373.95458984375 proc.cpu.threads‚17) disk./.usagePercent‚39.58850460018915 disk.dm-1.out‚ 72.94921875% disk.nvme0n1.out‚7373.95458984375 disk.rbd1.in‚0 disk.sda.in‚0 cpu‚25.938280101687575 network.recv‚904284 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚95.45795917510986* proc.memory.percent‚16.916823387145996 disk.sda3.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 4355.39453125 disk.sda1.in‚0( proc.memory.availableMB‚ 744.16796875# proc.memory.rssMB‚ 2771.65234375 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 4355.39453125ÖotØ“´‚(:ˆ ˆíÒùó—ï gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§¤cĵ‚(:¹ ˆíÒ­¯•ñ' disk.nvme0c0n1.out‚7378.53076171875 cpu‚25.937314288247954 disk.dm-1.out‚ 73.37890625 disk.rbd0.in‚0! disk.nvme0n1.in‚ 4355.39453125% disk.nvme0n1.out‚7378.53076171875" disk.dm-0.out‚7378.53076171875 disk.sda.in‚0# proc.memory.rssMB‚ 2771.65234375$ disk./.usageGB‚589.8957595825195 disk.rbd0.out‚0 disk.dm-0.in‚ 4355.39453125 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.58839938109257 disk.drbd2165.in‚0# memory_percent‚96.9616174697876* proc.memory.percent‚16.916823387145996 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 73.37890625 network.sent‚ 6.147799e+06 disk.dm-1.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 4355.39453125( proc.memory.availableMB‚ 497.80859375 disk.sda3.out‚ 73.37890625 disk.sda2.out‚0 network.recv‚908634 disk.sda2.in‚0 disk.rbd1.out‚8.9375¾›©ž¶‚(:“ ˆíÒ³¹·î gpu.0.gpu‚61.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì*ÄíÁ·‚(:¶ ˆíÒˆî×ï$ disk./.usageGB‚589.8957557678223 disk.sda3.out‚ 73.7421875! disk.nvme0n1.in‚ 4355.39453125 disk.rbd1.out‚8.9375 disk.sda.in‚0 disk.sda.out‚ 73.7421875 disk.sda1.out‚0( proc.memory.availableMB‚ 497.66015625 disk.dm-1.in‚0 disk.drbd2165.out‚0 cpu‚25.93614377121184) disk./.usagePercent‚39.58839912508503 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚7384.46435546875 disk.dm-1.out‚ 73.7421875% disk.nvme0n1.out‚7384.46435546875 disk.dm-0.in‚ 4355.39453125 disk.sda1.in‚0# proc.memory.rssMB‚ 2771.65234375* proc.memory.percent‚16.916823387145996 disk.rbd0.in‚0 disk.sda2.in‚0" disk.dm-0.out‚7384.46435546875$ memory_percent‚96.96252346038818 proc.cpu.threads‚17 network.sent‚ 6.150864e+06 disk.sda3.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 4355.39453125 network.recv‚911968 disk.sda2.out‚0 disk.rbd1.in‚0=¹“2¸- _wandbruntime‚3383  _runtime‚3383× -NŹ¿ _step‚317 % lr-AdamW/pg1‚0.000537442526695545 % lr-AdamW/pg2‚0.000537442526695545 trainer/global_step‚3899 % _timestamp‚1.7843661045183587e+09 _runtime‚3383.024340729wÕ¹ÊºÄ _step‚317 % lr-AdamW/pg1‚0.000537442526695545 % lr-AdamW/pg2‚0.000537442526695545 trainer/global_step‚3899 % _timestamp‚1.7843661045183587e+09 _runtime‚3383.024340729½Iç Ù2»- _wandbruntime‚3383  _runtime‚3383üŽV …¼ÚO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8378171 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30854654 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30854656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36076969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4114438984192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚595756736724480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚595756378345984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4158800272384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592713010151115 MGDeviceStatsMonitor.on_train_buÇqÞ[atch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378345984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4114780323328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086946 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130639327422464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚591641956401152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4114438984192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4158884655104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134798127694848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4108494696397 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941602ÐD~³å'½ßO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚591641956401152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4114438984192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4158884655104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚591641939361792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134798127694848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4108494696397 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941602 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8378171 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30854654 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30854656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36076969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4114438984192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3899 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚595756736724480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚595756378345984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4158800272384 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚592713010151115 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6834058 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚595756378345984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4114780323328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30854654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36086946 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130639327422464 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360¾Õ~‹2¾- _wandbruntime‚3383  _runtime‚33838`<çê&¿äM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚592860531876709 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36096012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134832575100416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36086021 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6835803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚595904632002560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚595904990388224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948307 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚591789180391424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588751383715840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30862288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30862286 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30862286 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36086021 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15216239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130672774793216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4159907489280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3899 % _timestamp‚1.7843661051732147e+09 _runtime‚3383.679338205 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚591789197430784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115792957440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592860889466071 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30862288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15215977 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4109505750231 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6835806 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130672756180992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4159818919424üæØNÀéM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚591789197430784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚595904632002560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚595904990388224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948307 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚591789180391424 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588751383715840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30862288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30862286 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30862286 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36086021 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15216239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130672774793216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batcº:H¸nh_end/active_bytes.all.current‚ 358385664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4159907489280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3899 % _timestamp‚1.7843661051732147e+09 _runtime‚3383.679338205 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚591789197430784 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4115792957440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚592860889466071 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30862288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15215977 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4109505750231 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6835806 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958300 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130672756180992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4159818919424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚592860531876709 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚591789180391424 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 _step‚319 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36096012 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134832575100416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36086021 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6835803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948307 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262¿ÁËÛÀ2Á- _wandbruntime‚3383  _runtime‚3383…S”É Âƒ /loss/FBt1/detection_loss‚0.4203312193230528 epoch‚9 2loss/FBt2/assignment_loss‚0.026704868213966962 /loss/FRt2/detection_loss‚1.5003312719029267 1loss/SRqqt1/detection_loss‚0.9162223547968545 %loss/total_loss‚9.402294816272235 _runtime‚3383.680488134 /loss/FBt2/detection_loss‚1.0503619680137828 % _timestamp‚1.7843661051748154e+09 _step‚320 0loss/FRt1/assignment_loss‚2.0809547768698797 2loss/SRqqt1/assignment_loss‚0.5583407586977274 2loss/FBt1/assignment_loss‚0.014368282316616514 /loss/FRt1/detection_loss‚0.5871090026981224 trainer/global_step‚3899 0loss/FRt2/assignment_loss‚1.0334335587547037 2loss/SRqqt2/assignment_loss‚0.5516918683838885 1loss/SRqqt2/detection_loss‚0.6624448863007129Uä³ÜÈ epoch‚9 2loss/FBt2/assignment_loss‚0.026704868213966962 /loss/FRt2/detection_loss‚1.5003312719029267 1loss/SRqqt1/detection_loss‚0.9162223547968545 %loss/total_loss‚9.402294816272235 _runtime‚3383.680488134 /loss/FBt2/detection_loss‚1.0503619680137828 % _timestamp‚1.7843661051748154e+09 _step‚320 0loss/FRt1/assignment_loss‚2.0809547768698797 2loss/SRqqt1/assignment_loss‚0.5583407586977274 2loss/FBt1/assignment_loss‚0.014368282316616514 /loss/FRt1/detection_loss‚0.5871090026981224 trainer/global_step‚3899 0loss/FRt2/assignment_loss‚1.0334335587547037 2loss/SRqqt2/assignment_loss‚0.5516918683838885 1loss/SRqqt2/detection_loss‚0.6624448863007129 /loss/FBt1/detection_loss‚0.4203312193230528Àx2C‘“Ä‚(:ˆ ŸˆíÒÔçíí gpu.0.gpu‚81.0 gpu.0.memory‚22+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßkv†¯Å‚(:¤ ŸˆíÒ¿ÄËï disk.sda1.out‚0 network.sent‚ 6.1558e+06 network.recv‚915355 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 4455.8515625) disk./.usagePercent‚39.58841141344668 disk.dm-1.out‚ 74.515625 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 74.515625 disk.sda1.in‚0$ disk.nvme0n1.out‚7387.4892578125 disk.rbd1.out‚9( proc.memory.availableMB‚ 594.39453125# disk./.usageGB‚589.895938873291" proc.memory.rssMB‚ 2642.7578125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚7387.4892578125 disk.sda.in‚0 disk.dm-0.in‚ 4455.8515625 cpu‚25.935034813531406 disk.nvme0n1.in‚ 4455.8515625 disk.rbd1.in‚0$ memory_percent‚96.37210369110107) proc.memory.percent‚16.13011360168457 disk.sda3.out‚ 74.515625! disk.dm-0.out‚7387.4892578125eþ´õžÆ‚(:“ ®ˆíÒæíîî gpu.0.gpu‚95.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚40.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ !°Ç‚(:¥ ®ˆíÒô¨•ñ disk.dm-0.in‚ 4455.8515625' proc.memory.availableMB‚ 594.1484375 network.sent‚ 6.219019e+06 disk.sda1.in‚0 disk.dm-0.out‚7392.115234375 network.recv‚919738 disk.sda.out‚ 74.8828125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚9% disk.nvme0c0n1.out‚7392.115234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚96.37360572814941) proc.memory.percent‚16.13011360168457 disk.drbd2165.out‚0$ disk./.usageGB‚589.8964500427246 disk.sda.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.58844571845626 disk.sda3.in‚0 disk.sda3.out‚ 74.8828125 disk.nvme0n1.in‚ 4455.8515625# disk.nvme0n1.out‚7392.115234375 disk.sda2.in‚0" proc.memory.rssMB‚ 2642.7578125 cpu‚25.93398214076353 disk.dm-1.out‚ 74.8828125" disk.nvme0c0n1.in‚ 4455.85156250ñ‘È‚(:† ½ˆíÒ‚‡©í gpu.0.gpu‚72.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚132.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òmt¼É‚(:± ½ˆíÒä«Žï disk.dm-1.out‚ 75.31640625 disk.dm-0.in‚ 4455.8515625& disk.nvme0c0n1.out‚7396.1904296875 disk.sda.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 4455.8515625 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd1.out‚9.0625( proc.memory.availableMB‚ 593.54296875 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 75.31640625 disk.sda1.in‚0 disk.sda3.out‚ 75.31640625 network.recv‚923202( disk./.usagePercent‚39.5884452064412$ disk./.usageGB‚589.8964424133301 disk.sda2.out‚0 disk.sda1.out‚0) proc.memory.percent‚16.13011360168457 proc.cpu.threads‚17 network.sent‚ 6.223864e+06 disk.drbd2165.out‚0$ disk.nvme0n1.out‚7396.1904296875 disk.sda2.in‚0 cpu‚25.932995454450626 disk.nvme0n1.in‚ 4455.8515625! disk.dm-0.out‚7396.1904296875 disk.rbd1.in‚0$ memory_percent‚96.37730121612549" proc.memory.rssMB‚ 2642.7578125ïêp2Ê- _wandbruntime‚3423  _runtime‚3423FmÍÆËÀ $ _timestamp‚1.784366145457583e+09 _runtime‚3423.963549409 _step‚321 & lr-AdamW/pg1‚0.0005361941146840189 & lr-AdamW/pg2‚0.0005361941146840189 trainer/global_step‚39498lÎËÌÅ & lr-AdamW/pg1‚0.0005361941146840189 & lr-AdamW/pg2‚0.0005361941146840189 trainer/global_step‚3949 $ _timestamp‚1.784366145457583e+09 _runtime‚3423.963549409 _step‚321ÁØ€j2Í- _wandbruntime‚3423  _runtime‚3423¢Ú:à'ÎÚO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚603169483707904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4165134486016 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4209906721280 _runtime‚3423.967723953 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599003990843392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600089517820873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36531421 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31236256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595930406637568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599004007882752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136520690682368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4165134486016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132310783961088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4209811852800 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6921311 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8481150 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603169483707904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36541398 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31236254 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15402461 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3949 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125329408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚599004007882752 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3949 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚603169125329408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚599004007882752 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚603169483707904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4165134486016 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4209906721280 _runtime‚3423.967723953 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚599003990843392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚600089517820873 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36531421 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31236256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595930406637568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚599004007882752 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚136520690682368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4165134486016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31236256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132310783961088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4209811852800 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6921311 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8481150 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚603169483707904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36541398 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31236254 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15402461 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8482980 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36550464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚599151231873024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36550464 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132344212719616 trainer/global_step‚3949 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4159781687137 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.larg´ßxîée_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540473 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36540473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600237039546579 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15406033 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4210923018240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4166488459264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599151231873024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136555015972864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132344194107392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31243888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8483264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7843661460882752e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 _step‚323 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6923056 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67784359 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6923053 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚599151248912384 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚284 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8482980 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36550464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚287 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚599151231873024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36550464 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132344212719616 trainer/global_step‚3949 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4159781687137 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36540473 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36540473 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600237039546579 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15406033 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4210923018240 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4166488459264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚599151231873024 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136555015972864 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132344194107392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31243888 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993ï´Ê¹2Ó- _wandbruntime‚3424  _runtime‚3424fïÑÔÿ /loss/FBt2/detection_loss‚1.1673183330410235 trainer/global_step‚3949 _runtime‚3424.595069776 0loss/FRt2/assignment_loss‚1.0205979413057427 epoch‚9 0loss/FRt1/assignment_loss‚1.9284247468514155 2loss/SRqqt1/assignment_loss‚0.6192591607130173 1loss/FBt2/assignment_loss‚0.04736830755711702 /loss/FRt1/detection_loss‚0.6110830120375271 _step‚324 2loss/SRqqt2/assignment_loss‚0.5225899960051954 1loss/FBt1/assignment_loss‚0.01909743085130726 /loss/FBt1/detection_loss‚0.7054421752575816 % _timestamp‚1.7843661460893242e+09 %loss/total_loss‚9.973055514936853 /loss/FRt2/detection_loss‚1.3902888007061818 /loss/SRqqt1/detection_loss‚1.22235920234249 1loss/SRqqt2/detection_loss‚0.7192264082682538)ÈN© Õ„ 2loss/SRqqt1/assignment_loss‚0.6192591607130173 1loss/FBt2/assignment_loss‚0.04736830755711702 /loss/FRt1/detection_loss‚0.6110830120375271 _step‚324 2loss/SRqqt2/assignment_loss‚0.5225899960051954 1loss/FBt1/assignment_loss‚0.01909743085130726 /loss/FBt1/detection_loss‚0.7054421752575816 % _timestamp‚1.7843661460893242e+09 %loss/total_loss‚9.973055514936853 /loss/FRt2/detection_loss‚1.3902888007061818 /loss/SRqqt1/detection_loss‚1.22235920234249 1loss/SRqqt2/detection_loss‚0.7192264082682538 /loss/FBt2/detection_loss‚1.1673183330410235 trainer/global_step‚3949 _runtime‚3424.595069776 0loss/FRt2/assignment_loss‚1.0205979413057427 epoch‚9 0loss/FRt1/assignment_loss‚1.9284247468514155Ä^›’Ö‚(:’ ̈íÒÊŽí gpu.0.gpu‚44.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ¼y“Ú‚(:ˆ êˆíÒî°›ï gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯îPå¾Û‚(:³ êˆíÒêäðï disk.sda3.out‚ 76.18359375 disk.dm-1.out‚ 76.18359375 disk.drbd2165.out‚0% disk.nvme0n1.out‚7413.38623046875 disk.sda2.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2642.7578125 disk.rbd1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 network.sent‚ 6.295907e+06 network.recv‚933403 disk.sda2.out‚0 disk.rbd0.in‚0$ memory_percent‚96.37439250946045) proc.memory.percent‚16.13011360168457 proc.cpu.threads‚17$ disk./.usageGB‚589.8966293334961 disk.sda.out‚ 76.18359375 disk.rbd0.out‚0 cpu‚25.92983596625248 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 4570.1015625( proc.memory.availableMB‚ 594.01953125) disk./.usagePercent‚39.58845775081038" disk.dm-0.out‚7413.38623046875' disk.nvme0c0n1.out‚7413.38623046875 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 4570.1015625 disk.rbd1.out‚9.125" disk.nvme0c0n1.in‚ 4570.1015625%Ò²f2Ü- _wandbruntime‚3465  _runtime‚3465_,nÇÝÁ trainer/global_step‚3999 % _timestamp‚1.7843661865681438e+09 _runtime‚3465.074089733 _step‚325 & lr-AdamW/pg1‚0.0005349299550420616 & lr-AdamW/pg2‚0.0005349299550420616/«½WÌÞÆ & lr-AdamW/pg2‚0.0005349299550420616 trainer/global_step‚3999 % _timestamp‚1.7843661865681438e+09 _runtime‚3465.074089733 _step‚325 & lr-AdamW/pg1‚0.0005349299550420616ÅQ¥+,2ß- _wandbruntime‚3465  _runtime‚3465wŸ!àÚO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4260902205440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901381 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _step‚326 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606366042324992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138243124092928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215829987840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7008558 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36995850 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36985873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4216171326976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15592390 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚3999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215829987840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607465310312131 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843661865692885e+09 _runtime‚3465.075426297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610582230691328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872312832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603255940231168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133982221887488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8583556 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610582230691328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872312832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603727 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚606366042324992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171326976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8583829 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_ 9ØQtrain_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138243016903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133982203275264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4209387120323 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31617856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4260813628416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995850 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7008561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚606366059364352u§~âå'áßO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚610581872312832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚603255940231168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133982221887488 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8583556 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚610582230691328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚610581872312832 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603727 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚606366042324992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4216171326976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8583829 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138243016903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133982203275264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4209387120323 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31617856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4260813628416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36995850 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7008561 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚606366059364352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4260902205440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚607465667901381 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 _step‚326 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚606366042324992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138243124092928 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215829987840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7008558 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36995850 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36985873 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4216171326976 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15592390 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 trainer/global_step‚3999 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215829987840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚607465310312131 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843661865692885e+09 _runtime‚3465.075426297 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚610582230691328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979Æ3Ö 2â- _wandbruntime‚3465  _runtime‚3465Ø™}Øê&ãäM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134015650646016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607613189627087 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37004916 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4261838583296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚606513300393984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36994925 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7010306 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138277573604864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚3999 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630404 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15595898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3465.707021877 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603402433863680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630404 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216842614784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620411 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8585595 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31625488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31625486 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134015632033792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37004916 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4261838583296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚606513300393984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36994925 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7010306 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7010303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138277573604864 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 trainer/global_step‚3999 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630404 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15595898 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚3465.707021877 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603402433863680 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630404 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216842614784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620411 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8585595 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31625488 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31625486 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134015632033792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134015650646016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚607613189627087 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,jet/accuracy_4_of_4‚0.009931244887411594 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17979909479618073 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923170804977417 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8164210319519043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04146698862314224 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2149629145860672 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1859053373336792 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25520020723342896 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07239922136068344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.627587080001831 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.40603819489479065 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33564814925193787 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5961756110191345 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18933579325675964 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3743864595890045 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20184767246246338 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011943300254642963 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.565416693687439 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02245425246655941 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8514905571937561 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951432347297668 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2466685026884079 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.6182594895362854 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14329929649829865 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939543604850769 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7644615173339844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7354792952537537 jet/accuracy_4_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18142087757587433 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6793595552444458 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6347578763961792 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6341111063957214 /particle/accuracy_4_of_4‚0.9386892914772034 'particle/f_score‚0.6315053701400757 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.565883219242096 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7569464445114136 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9643166661262512 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5396853685379028 epoch‚9 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942887425422668 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7310150861740112 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9969437718391418 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9957706928253174 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5528061985969543 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0400557704269886 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1765279918909073 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.519066333770752 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14433586597442627 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13278290629386902 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02113279327750206 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.683790922164917 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964209794998169 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.635148823261261 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8223734498023987 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6287720203399658 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16789104044437408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9957706928253174 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 'particle/accuracy‚0.845055878162384 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.802872359752655 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1178082525730133 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9938886761665344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6569007635116577 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922472834587097 +jet/accuracy_3_of_3‚0.06509509682655334 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.680277407169342 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7456727027893066 _runtime‚3482.882080856 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6262491941452026 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.555695116519928 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7356417775154114 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20200160145759583 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6226403117179871 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3736971318721771 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6111963391304016 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7268419861793518 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4440760016441345 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07799047231674194 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8211165070533752 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8991703987121582 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18058478832244873 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15538452565670013 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061977092176675797 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6769263744354248 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18903210759162903 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 *jet/accuracy_1_of_3‚0.9303457140922546 *jet/accuracy_2_of_4‚0.8095184564590454 particle/accuracy_1_of_1‚1 _step‚329 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 )loss/val_total_loss‚9.077027888357115 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8616669774055481 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7967119812965393 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6314842104911804 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.734937846660614 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2616221010684967 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6540552973747253 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061977092176675797 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.635148823261261 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16776929795742035 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7364667057991028 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18187077343463898 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9861899614334106 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9826451539993286 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3359375 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18903210759162903 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07778578251600266 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09558750689029694 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4019223749637604 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6369891166687012 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9214823842048645 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6999194025993347 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16969448328018188 particle/accuracy_1_of_2‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6930343508720398 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7888192534446716 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3999 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9832316637039185 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18868005275726318 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7322096824645996 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9907050132751465 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46966537833213806 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8242629766464233 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 jet/accuracy_3_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03232354670763016 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6640625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9934744238853455 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.14297406375408173 particle/accuracy_3_of_3‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6301212310791016 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9853219389915466 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4596904516220093 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6348386406898499 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7891405820846558 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7638355493545532 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.203121155500412 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 )jet/accuracy_1_of_2‚0.785554051399231 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3947898745536804 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 % _timestamp‚1.7843662043754642e+09 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07245389372110367 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20682984590530396ÏÄG«®9ìß{ 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9832316637039185 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945749640464783 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18868005275726318 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7322096824645996 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9907050132751465 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46966537833213806 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8242629766464233 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 jet/accuracy_3_of_5‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03232354670763016 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6640625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9934744238853455 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.14297406375408173 particle/accuracy_3_of_3‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6301212310791016 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9853219389915466 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4596904516220093 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6348386406898499 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7891405820846558 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7638355493545532 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.203121155500412 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 )jet/accuracy_1_of_2‚0.785554051399231 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3947898745536804 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 % _timestamp‚1.7843662043754642e+09 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07245389372110367 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20682984590530396 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8211165070533752 +particle/specificity‚0.9070800542831421 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29905277490615845 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9652905464172363 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19298982620239258 /particle/accuracy_3_of_4‚0.9889726042747498 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7340496778488159 6validation_average_jet_accuracy‚0.3591185212135315 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965887665748596 +jet/accuracy_1_of_1‚0.31370213627815247 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1273433417081833 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9093071222305298 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,jet/accuracy_4_of_4‚0.009931244887411594 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17979909479618073 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9923170804977417 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8164210319519043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04146698862314224 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2149629145860672 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1859053373336792 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.25520020723342896 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07239922136068344 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.627587080001831 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.40603819489479065 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.33564814925193787 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5961756110191345 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 particle/accuracy_2_of_2‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18933579325675964 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3743864595890045 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20184767246246338 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011943300254642963 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.565416693687439 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02245425246655941 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8514905571937561 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9951432347297668 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2466685026884079 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +particle/sensitivity‚0.6182594895362854 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 particle/accuracy_4_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14329929649829865 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_2_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939543604850769 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7644615173339844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7354792952537537 jet/accuracy_4_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.18142087757587433 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6793595552444458 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6347578763961792 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6341111063957214 /particle/accuracy_4_of_4‚0.9386892914772034 'particle/f_score‚0.6315053701400757 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.565883219242096 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7569464445114136 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9643166661262512 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.5396853685379028 epoch‚9 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942887425422668 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7310150861740112 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9969437718391418 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9957706928253174 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5528061985969543 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0400557704269886 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1765279918909073 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.519066333770752 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.14433586597442627 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13278290629386902 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02113279327750206 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.683790922164917 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964209794998169 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.635148823261261 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8223734498023987 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6287720203399658 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3906250298023224 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16789104044437408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9957706928253174 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 'particle/accuracy‚0.845055878162384 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.802872359752655 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1178082525730133 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9938886761665344 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6569007635116577 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9922472834587097 +jet/accuracy_3_of_3‚0.06509509682655334 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.680277407169342 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7456727027893066 _runtime‚3482.882080856 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6262491941452026 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.555695116519928 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7356417775154114 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20200160145759583 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6226403117179871 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3736971318721771 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6111963391304016 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7268419861793518 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4440760016441345 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07799047231674194 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8211165070533752 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8991703987121582 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18058478832244873 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15538452565670013 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.061977092176675797 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6769263744354248 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18903210759162903 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 *jet/accuracy_1_of_3‚0.9303457140922546 *jet/accuracy_2_of_4‚0.8095184564590454 particle/accuracy_1_of_1‚1 _step‚329 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 )loss/val_total_loss‚9.077027888357115 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8616669774055481 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7967119812965393 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6314842104911804 particle/accuracy_1_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.734937846660614 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2616221010684967 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6540552973747253 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.061977092176675797 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.635148823261261 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16776929795742035 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7364667057991028 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18187077343463898 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9861899614334106 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9826451539993286 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3359375 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18903210759162903 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07778578251600266 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09558750689029694 9"Purity/*FRt1SRq$Î97qt1FBt/event_purity‚0.4019223749637604 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6369891166687012 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9214823842048645 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6999194025993347 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16969448328018188 particle/accuracy_1_of_2‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6930343508720398 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7888192534446716 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚3999É  øží‚(:“ ˆ‰íÒ»å”î gpu.0.gpu‚77.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö˜a¹î‚(:® ˆ‰íÒ”õ¸ð network.recv‚941120 disk.sda3.out‚ 76.859375 disk.dm-1.out‚ 76.859375 disk.sda2.in‚0 cpu‚25.946193185542995 disk.nvme0n1.in‚ 4570.1015625% disk.nvme0n1.out‚7671.90673828125" disk.dm-0.out‚7671.90673828125" disk.nvme0c0n1.in‚ 4570.1015625 network.sent‚ 6.363708e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.sda1.in‚0* disk./.usagePercent‚39.588549401507635$ memory_percent‚95.47276496887207 proc.cpu.threads‚17$ disk./.usageGB‚589.8979949951172 disk.dm-0.in‚ 4570.1015625 disk.rbd1.out‚9.25 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚7671.90673828125 disk.sda.out‚ 76.859375 disk.sda1.out‚0' proc.memory.availableMB‚ 741.7421875* proc.memory.percent‚16.774511337280273 disk.sda3.in‚0" proc.memory.rssMB‚ 2748.3359375c7lžï‚(:“ —‰íÒùþ‡ï gpu.0.gpu‚41.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚258.298' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.659600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ví¸ð‚(:­ —‰íÒ¯‚Íï disk.sda2.out‚0' disk.nvme0c0n1.out‚7921.31103515625 disk.rbd0.out‚0 network.recv‚945548% disk.nvme0n1.out‚7921.31103515625 disk.sda2.in‚0" proc.memory.rssMB‚ 2748.3359375 cpu‚25.945090685523933 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 77.47265625) disk./.usagePercent‚39.58848258354119 disk.dm-0.in‚ 4669.0625 disk.rbd1.in‚0 disk.sda1.out‚0$ memory_percent‚95.46968936920166 network.sent‚ 6.421167e+06 disk.sda3.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚7921.31103515625 disk.sda3.out‚ 77.47265625 disk.dm-1.out‚ 77.47265625 disk.nvme0c0n1.in‚ 4669.0625 disk.sda.in‚0* proc.memory.percent‚16.774511337280273 disk.nvme0n1.in‚ 4669.0625$ disk./.usageGB‚589.8969993591309 disk.dm-1.in‚0 disk.rbd1.out‚9.3125 disk.sda1.in‚0( proc.memory.availableMB‚ 742.24609375 proc.cpu.threads‚17õîåñ‚(:’ ¦‰íÒÀƒî gpu.0.gpu‚88.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚265.603' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.12060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@äU´ò‚(:© ¦‰íÒó…¢ï$ disk./.usageGB‚589.8970718383789 disk.nvme0c0n1.in‚ 4669.0625 disk.rbd0.in‚0 disk.dm-0.in‚ 4669.0625" disk.dm-0.out‚7926.01611328125 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚39.58848744768435 disk.sda.in‚0 disk.sda1.in‚0 disk.sda.out‚ 77.7734375 disk.sda3.out‚ 77.7734375 disk.dm-1.out‚ 77.7734375 disk.rbd1.out‚9.3125 disk.sda1.out‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚7926.01611328125 network.sent‚ 6.426148e+06 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 741.51953125 cpu‚25.94391926745011 disk.sda2.in‚0 network.recv‚948699' disk.nvme0c0n1.out‚7926.01611328125 disk.rbd1.in‚0$ memory_percent‚95.47412395477295* proc.memory.percent‚16.774511337280273 disk.sda3.in‚0 disk.nvme0n1.in‚ 4669.0625" proc.memory.rssMB‚ 2748.33593753NW2ó- _wandbruntime‚3525  _runtime‚3525Ÿo¥ÇôÁ & lr-AdamW/pg1‚0.0005336501289642841 & lr-AdamW/pg2‚0.0005336501289642841 trainer/global_step‚4049 % _timestamp‚1.7843662467419574e+09 _runtime‚3525.247912247 _step‚330²{ vÌõÆ _step‚330 & lr-AdamW/pg1‚0.0005336501289642841 & lr-AdamW/pg2‚0.0005336501289642841 trainer/global_step‚4049 % _timestamp‚1.7843662467419574e+09 _runtime‚3525.247912247ʯ–åˆ2ö- _wandbruntime‚3525  _runtime‚3525;Oþß'÷ÙO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15818671 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32120122 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32120120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4284600036864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8720343 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32120122 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37520629 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615657660456960 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚331 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3525.252738969 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7098331 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 trainer/global_step‚4049 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919154688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37510652 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630772 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919154688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32120120 $ _timestamp‚1.784366246742976e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619942277533184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4284258697728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139999901776896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619942277533184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615657677496320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139999798781952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4278063882355 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921203 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615657660456960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37510652 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4284258697728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4330245480448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4277723332465 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135669571913728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612490137038848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4330329863168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32120122 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37520629 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚615657660456960 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚331 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚3525.252738969 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7098331 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚612490119999488 trainer/global_step‚4049 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚619941919154688 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37510652 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630772 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚619941919154688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32120120 $ _timestamp‚1.784366246742976e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚619942277533184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4284258697728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139999901776896 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚619942277533184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚615657677496320 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚139999798781952 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4278063882355 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616768200921203 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚615657660456960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37510652 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4284258697728 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4330245480448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4277723332465 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135669571913728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚612490137038848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4330329863168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37519704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612636630671360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4285612670976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15822400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4285612670976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7100073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657449 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚4049 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612636647710720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531196928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8722327 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140034254582272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279074936189 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚3525.880030491 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331354800640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚332 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615804901486592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615804901486592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batchñçß t _end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616915365057547 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127754 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37529695 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271324672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37529695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140034355472896 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37519704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32127752ñ= ~ï&ûéM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8722597 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15822673 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135702982060032 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657449 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚4049 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚612636647710720 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚615804918525952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚620090531196928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8722327 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140034254582272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4279074936189 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 _runtime‚3525.880030491 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4331354800640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _step‚332 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚615804901486592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚615804901486592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69647456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616915365057547 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32127754 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37529695 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4285271324672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37529695 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140034355472896 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37519704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32127752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4331272522240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚616915722646909 % _timestamp‚1.7843662473738823e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37519704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚612636630671360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4285612670976 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15822400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4285612670976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7100073 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476803 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16010751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4381474687488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624143993412461 _runtime‚3566.296740933 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141722465915392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚619815670632448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32501720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4334954199552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37965104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141722579396096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚335 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37965104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚623019728977920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16010481 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954199552 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666138112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚624144351001711 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚627354666138112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚624144351001711 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476803 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16010751 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.onŸ‚[_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4381474687488 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚624143993412461 _runtime‚3566.296740933 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚141722465915392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚623019711938560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚619815670632448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32501720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4334954199552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37965104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141722579396096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 _step‚335 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37965104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚623019728977920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16010481 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4334954199552 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493501 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37974156 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7187323 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141757033573376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16014265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32509352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32509354 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚623166970007552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4382512696320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624291515138055 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32509352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37974156 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493501 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37984147 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚624291872727417 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329350873095 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623166952968192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0®!â>î&‹èM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37974156 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70493501 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37984147 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚624291872727417 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4329350873095 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚623166952968192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚137374438598656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4329691423097 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493501 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37974156 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7187323 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141757033573376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16014265 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32509352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32509354 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚623166970007552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4382512696320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚624291515138055 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32509352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2Ð@<:2Œ- _wandbruntime‚3566  _runtime‚3566ÊÕ\ /loss/FRt1/assignment_loss‚2.042486261605747 /loss/FRt1/detection_loss‚0.5725877321230197 /loss/FRt2/detection_loss‚1.4796774616887374 1loss/SRqqt2/detection_loss‚0.6400662979537171 _runtime‚3566.932822854 1loss/FBt1/assignment_loss‚0.04077347989562338 0loss/SRqqt1/detection_loss‚0.894919860512828 trainer/global_step‚4099 0loss/FRt2/assignment_loss‚0.8960583386689984 2loss/SRqqt1/assignment_loss‚0.6982970243033892 2loss/SRqqt2/assignment_loss‚0.5437440232876101 /loss/FBt2/detection_loss‚0.7535171998555169 % _timestamp‚1.7843662884271455e+09 _step‚337 2loss/FBt2/assignment_loss‚0.003506319218047129 /loss/FBt1/detection_loss‚0.5861257855705917 %loss/total_loss‚9.151759784683826 epoch‚10xtk¿ Ž† /loss/FRt1/detection_loss‚0.5725877321230197 /loss/FRt2/detection_loss‚1.4796774616887374 1loss/SRqqt2/detection_loss‚0.6400662979537171 _runtime‚3566.932822854 1loss/FBt1/assignment_loss‚0.04077347989562338 0loss/SRqqt1/detection_loss‚0.894919860512828 trainer/global_step‚4099 0loss/FRt2/assignment_loss‚0.8960583386689984 2loss/SRqqt1/assignment_loss‚0.6982970243033892 2loss/SRqqt2/assignment_loss‚0.5437440232876101 /loss/FBt2/detection_loss‚0.7535171998555169 % _timestamp‚1.7843662884271455e+09 _step‚337 2loss/FBt2/assignment_loss‚0.003506319218047129 /loss/FBt1/detection_loss‚0.5861257855705917 %loss/total_loss‚9.151759784683826 epoch‚10 /loss/FRt1/assignment_loss‚2.042486261605747ÑšiRL‘‚(:† Ó‰íÒœùî gpu.0.gpu‚87.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚235.14' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨í&n¿‚(:´ Ó‰íÒÈâï disk.sda2.in‚0* proc.memory.percent‚16.774511337280273 disk.dm-0.in‚ 4785.09765625' disk.nvme0c0n1.out‚7941.17529296875 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 4785.09765625" proc.memory.rssMB‚ 2748.3359375$ disk./.usageGB‚589.8967781066895 disk.dm-1.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚7941.17529296875 proc.cpu.threads‚17 network.sent‚ 6.49881e+06) disk./.usagePercent‚39.58846773510421 disk.dm-1.out‚ 78.63671875" disk.dm-0.out‚7941.17529296875 cpu‚25.940788730198875 disk.sda3.in‚0 disk.rbd1.out‚9.4375$ proc.memory.availableMB‚741.9375 network.recv‚959868 disk.sda3.out‚ 78.63671875 disk.rbd0.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4785.09765625 disk.sda.out‚ 78.63671875$ memory_percent‚95.47157287597656 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0º‰.¥’‘‚(:‡ â‰íÒ×»­ï gpu.0.gpu‚75.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍæÙŸº’‚(:¯ â‰íÒ—‡¤ñ disk.sda2.out‚0# disk.nvme0c0n1.in‚ 4785.09765625$ memory_percent‚95.47157287597656 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚9.4375' disk.nvme0c0n1.out‚7947.22509765625 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.774511337280273! disk.nvme0n1.in‚ 4785.09765625% disk.nvme0n1.out‚7947.22509765625 disk.sda1.in‚0$ proc.memory.availableMB‚741.9375 proc.cpu.threads‚17 disk.dm-0.in‚ 4785.09765625 disk.rbd1.in‚0 cpu‚25.939765493268215) disk./.usagePercent‚39.58845749480284 disk.sda3.out‚ 78.953125 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.out‚ 78.953125 disk.rbd0.out‚0$ disk./.usageGB‚589.8966255187988 disk.rbd0.in‚0" disk.dm-0.out‚7947.22509765625 network.sent‚ 6.567836e+06 network.recv‚971756 disk.dm-1.out‚ 78.953125" proc.memory.rssMB‚ 2748.3359375‰LI““‚(:ˆ ñ‰íÒ¤€¬î gpu.0.gpu‚34.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.231' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óz§p¾”‚(:³ ñ‰íÒ»À˜ï network.recv‚975116 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2748.3359375$ disk./.usageGB‚589.8971710205078 disk.sda.in‚0 disk.sda.out‚ 79.2734375 disk.dm-1.in‚0* proc.memory.percent‚16.774511337280273 disk.rbd1.in‚0 disk.drbd2165.out‚0$ memory_percent‚95.47011852264404 cpu‚25.938873763861068# disk.nvme0c0n1.in‚ 4785.09765625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 4785.09765625 disk.sda2.in‚0( proc.memory.availableMB‚ 742.17578125 disk.dm-0.in‚ 4785.09765625 disk.sda3.in‚0 disk.sda3.out‚ 79.2734375 disk.dm-1.out‚ 79.2734375 proc.cpu.threads‚17) disk./.usagePercent‚39.58849410388024 disk.rbd0.in‚0 disk.rbd1.out‚9.4375& disk.nvme0c0n1.out‚7950.7294921875 disk.sda1.out‚0 network.sent‚ 6.570906e+06$ disk.nvme0n1.out‚7950.7294921875! disk.dm-0.out‚7950.7294921875æd2•- _wandbruntime‚3607  _runtime‚3607„˜žuÆ–À & lr-AdamW/pg1‚0.0005310438073055656 & lr-AdamW/pg2‚0.0005310438073055656 trainer/global_step‚4149 % _timestamp‚1.7843663286772382e+09 _runtime‚ 3607.18479786 _step‚338~æ‰ûË—Å trainer/global_step‚4149 % _timestamp‚1.7843663286772382e+09 _runtime‚ 3607.18479786 _step‚338 & lr-AdamW/pg1‚0.0005310438073055656 & lr-AdamW/pg2‚0.0005310438073055656ÒdI{2˜- _wandbruntime‚3607  _runtime‚3607DÙ!C÷™ÚO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4378956306281 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492969 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302876 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4432716837376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚339 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚631520501082219 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16201596 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4385649701376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8929020 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991040512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7272828 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429533 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143445164603904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843663286779842e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419556 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649701376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4432628260352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4149 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16201851 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771500032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991040512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413121536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚634767771500032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429533 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8928768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627141187186688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚634767413121536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627141204226048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32883320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4378956306281 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚631520143492969 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302876 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4432716837376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚339 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139012447766528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚631520501082219 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16201596 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32883322 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4385649701376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8929020 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385991040512 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630381780459520 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7272828 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429533 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7272831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143445164603904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843663286779842e+09 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419556 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32883320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385649701376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4432628260352 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4149 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16201851 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634767771500032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385991040512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634767413121536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚634767771500032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38429533 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8928768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139012429154304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627141187186688 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚634767413121536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32883322 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627141204226048 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÓçÐ12›- _wandbruntime‚3607  _runtime‚3607ͧú}ê&œäM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32890954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16205609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139045876525056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7274573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433642639360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32890952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚631667665218563 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16205349 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚627287714897920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7274576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38438599 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚4149 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143479500552192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚634916025163776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025163776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143479618220032 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38438599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚631668022807925 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379967360003 _runtime‚3607.816161211 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329553 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚340 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32890954 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0LÜ‹Uó éM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38428608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 117667840 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4433741694976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚340 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32890954 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32890954 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139045857912832 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16205609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139045876525056 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7274573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4433642639360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 JDDeviceStatsMonitor¬Sº°ü.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32890952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚99055616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚631667665218563 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16205349 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚627287714897920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7274576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚630529021489152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38438599 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚4149 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143479500552192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚634916025163776 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚630529021489152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634916025163776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143479618220032 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38438599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚631668022807925 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4379967360003 _runtime‚3607.816161211 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329553 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9033297 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚638896293573477 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4429572793189 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72148907 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4429913343079 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚642180518483456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145167691302400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638896651162727 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634466737819648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4483824221696 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38883985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436345203200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160104960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16393101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784366369759394e+09 _runtime‚3648.265511212 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38874008CºÖŠy#¬ÞO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚637743814901760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145167800588800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚4199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚343 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚637743831941120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9033023 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9033297 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚638896293573477 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚637743831941120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4429572793189 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72148907 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚140683867080704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4429913343079 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚642180518483456 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145167691302400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138928 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚638896651162727 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634466737819648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4483824221696 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38883985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_staxœ€rkrt/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4436345203200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642180160104960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16393101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.784366369759394e+09 _runtime‚3648.265511212 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38874008×1ÇæL2­- _wandbruntime‚3648  _runtime‚3648âU&oê&®äM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843663703899605e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚3648.896213056 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7361823 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33272552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72155612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642328413761536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155612 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145202249486336 trainer/global_step‚4199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚642328772147200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16396878 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4430583846911 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4484935034880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38883060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9035322 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642328772147200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38893051 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888433 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚637891072970752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437357830144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚140717295839232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33272552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639043815299071 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7361826 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38883060 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38893051 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4484842270720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699176448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4430924396913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚637891055931392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚637891055931392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437357830144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ê$3ï&¯éM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642328772147200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38893051 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639044172888433 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚637891072970752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4437357830144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚140717295839232 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33272552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639043815299071 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7361826 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38883060 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38893051 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4484842270720 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634613248491520 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4437699176448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4430924396913 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚637891055931392 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚637891055931392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33272554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4437357830144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 % _timestamp‚1.7843663703899605e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚3648.896213056 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72165605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7361823 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33272552 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72155612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642328413761536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72155612 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145202249486336 trainer/global_step‚4199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚642328772147200 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16396878 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4430583846911 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4484935034880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38883060 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9035322 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ØÎïª2°- _wandbruntime‚3648  _runtime‚3648ƒ¹±€ 2loss/SRqqt2/assignment_loss‚0.5239905432761744 .loss/FRt1/detection_loss‚0.623956198573065 /loss/FRt2/detection_loss‚1.3649237968676584 %loss/total_loss‚9.366813834504022 2loss/FBt2/assignment_loss‚0.015627850805909473 1loss/SRqqt1/detection_loss‚1.0040526890968398 1loss/SRqqt2/detection_loss‚0.6186768958488336 .loss/FBt1/detection_loss‚0.526364883380912 0loss/FRt1/assignment_loss‚2.0496670382875632 .loss/FBt2/detection_loss‚1.069212738333679 trainer/global_step‚4199 % _timestamp‚1.7843663703910556e+09 0loss/FRt2/assignment_loss‚0.9868189728870167 1loss/FBt1/assignment_loss‚0.02576639467436525 epoch‚10 _runtime‚3648.896828228 _step‚345 2loss/SRqqt1/assignment_loss‚0.5577558324720054tº ²… 0loss/FRt2/assignment_loss‚0.9868189728870167 1loss/FBt1/assignment_loss‚0.02576639467436525 epoch‚10 _runtime‚3648.896828228 _step‚345 2loss/SRqqt1/assignment_loss‚0.5577558324720054 2loss/SRqqt2/assignment_loss‚0.5239905432761744 .loss/FRt1/detection_loss‚0.623956198573065 /loss/FRt2/detection_loss‚1.3649237968676584 %loss/total_loss‚9.366813834504022 2loss/FBt2/assignment_loss‚0.015627850805909473 1loss/SRqqt1/detection_loss‚1.0040526890968398 1loss/SRqqt2/detection_loss‚0.6186768958488336 .loss/FBt1/detection_loss‚0.526364883380912 0loss/FRt1/assignment_loss‚2.0496670382875632 .loss/FBt2/detection_loss‚1.069212738333679 trainer/global_step‚4199 % _timestamp‚1.7843663703910556e+09Ùà[Ÿ³‚(:’ ­ŠíÒÞŒÄï gpu.0.gpu‚44.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PKïÁ´‚(:¶ ­ŠíÒ“ÇÕñ disk.sda3.out‚ 80.61328125 disk.rbd0.in‚0! disk.dm-0.out‚7969.4716796875 disk.sda2.in‚0" proc.memory.rssMB‚ 2748.3359375 disk.rbd1.in‚0 disk.rbd1.out‚9.5625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚96.97823524475098 network.sent‚ 6.664922e+06* disk./.usagePercent‚39.588331795103535$ disk./.usageGB‚589.8947525024414 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚25.935083259952403 proc.cpu.threads‚17 network.recv‚997585# disk.nvme0c0n1.in‚ 4912.62890625& disk.nvme0c0n1.out‚7969.4716796875 disk.sda.out‚ 80.61328125 disk.drbd2165.in‚0$ disk.nvme0n1.out‚7969.4716796875' proc.memory.availableMB‚ 495.0859375* proc.memory.percent‚16.774511337280273! disk.nvme0n1.in‚ 4912.62890625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 80.61328125 disk.dm-0.in‚ 4912.62890625Ãóï„žµ‚(:“ ¼ŠíÒ£ë³ï gpu.0.gpu‚46.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚228.134' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.626799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ÐÇ<ö‚(:¸ ¼ŠíÒõ¼ñ disk.drbd2165.in‚0$ memory_percent‚96.97673320770264# disk.nvme0c0n1.in‚ 4912.62890625 disk.rbd1.in‚0 disk.dm-1.out‚ 80.953125" disk.dm-0.out‚7974.17236328125" proc.memory.rssMB‚ 2748.3359375' disk.nvme0c0n1.out‚7974.17236328125) disk./.usagePercent‚39.58834152338983$ disk./.usageGB‚589.8948974609375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚7974.17236328125( proc.memory.availableMB‚ 495.33203125 disk.dm-1.in‚0* proc.memory.percent‚16.774511337280273 cpu‚25.93406517739628 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 80.953125 network.recv‚ 1.002053e+06 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 6.717622e+06 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-0.in‚ 4912.62890625 disk.rbd1.out‚9.6875 disk.sda3.out‚ 80.953125 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 4912.62890625$g¡92·- _wandbruntime‚3689  _runtime‚3689O8¸JƸÀ & lr-AdamW/pg2‚0.0005283758192937909 trainer/global_step‚4249 $ _timestamp‚1.784366410968978e+09 _runtime‚3689.474982375 _step‚346 & lr-AdamW/pg1‚0.0005283758192937909y­³Ë¹Å $ _timestamp‚1.784366410968978e+09 _runtime‚3689.474982375 _step‚346 & lr-AdamW/pg1‚0.0005283758192937909 & lr-AdamW/pg2‚0.0005283758192937909 trainer/global_step‚4249Ú¦“íG2º- _wandbruntime‚3689  _runtime‚3689³—+¸#»ÚO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4535089730560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚645105883422720 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646272801243235 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487040705024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146890413349888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚649592907088384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33646520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142355305007104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚649593265466880 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382044160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39328460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9137353 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907088384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7447328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39338437 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645105866383360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16584681 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4480529829987 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4480189280097 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39328460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocatH•ˆ€½ ed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚641792254373888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146890308257792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚649593265466880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33646520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142355323619328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4535003250688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4535003250688 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487382044160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39328460 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9137353 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚649592907088384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33646522 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7447328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39338437 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645105866383360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16584681 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4480529829987 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4480189280097 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39328460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚641792254373888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146890308257792 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968Ûî “hž½‚(:“ ËŠíÒëÑí gpu.0.gpu‚86.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚254.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.962799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Æäƾ‚(:» ËŠíҚƀï disk.sda2.in‚0' disk.nvme0c0n1.out‚7977.53955078125* proc.memory.percent‚16.774511337280273% disk.nvme0n1.out‚7977.53955078125 cpu‚25.933021645156554 disk.rbd0.out‚0" disk.dm-0.out‚7977.53955078125# disk.nvme0c0n1.in‚ 4977.26953125 network.sent‚ 6.720544e+06 disk.drbd2165.out‚0 disk.dm-0.in‚ 4977.26953125 disk.rbd1.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 495.0703125 disk.sda1.in‚0 disk.dm-1.in‚0 network.recv‚ 1.005695e+06! disk.nvme0n1.in‚ 4977.26953125 disk.sda.in‚0 disk.sda.out‚ 81.4296875 disk.sda1.out‚0 disk.sda3.out‚ 81.4296875 disk.dm-1.out‚ 81.4296875) disk./.usagePercent‚39.58842984598914 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.out‚9.6875$ memory_percent‚96.97833061218262$ disk./.usageGB‚589.8962135314941 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2748.3359375 proc.cpu.threads‚17µíçŸ2¿- _wandbruntime‚3690  _runtime‚3690Ý(5ûè&ÀâM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚4249 _runtime‚3690.104820113 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968941 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39347503 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚348 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053331968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33654152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146924772379136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4536118794752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488394678272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16588681 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9139348 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001657 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚641938765045760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053331968 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33654154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646419965379579 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645253124452352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394678272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337512 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7449076 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33654152 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16588421 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33654154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4481540883821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784366411598645e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347503 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856á±įÁçM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33654154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚646419965379579 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645253124452352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488394678272 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39337512 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7449076 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33654152 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16588421 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33654154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4481540883821 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784366411598645e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39347503 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142388752377856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚4249 _runtime‚3690.104820113 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚646420322968941 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645253124452352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39347503 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B ‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚348 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4488053331968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33654152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7449073 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146924772379136 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4536118794752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488394678272 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16588681 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9139348 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚641938782085120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001657 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚641938765045760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4488053331968 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚653648593734493 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821011 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077545984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649117787967488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537736206848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39792889 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9240823 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657005654071808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16775140 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39782912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 trainer/global_step‚4299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16775404 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4586108185088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34028120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4531146316895 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657006012450304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7534581 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077545984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148612851118592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654071808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649117805006848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39782912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4586194664960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3730.397981542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39792889•°w‹å'ÎßO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34028120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144026742933504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4531146316895 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚657006012450304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7534581 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚652467917864960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4538077545984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚148612851118592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚657005654071808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚649117805006848 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39782912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7534578 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4586194664960 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3730.397981542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39792889 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9240562 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚653648593734493 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821011 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144026761545728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4538077545984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚649117787967488 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4537736206848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39792889 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9240823 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚657005654071808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16775140 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39782912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 trainer/global_step‚4299 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16775404 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4586108185088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664ßâiÝ2Ï- _wandbruntime‚3731  _runtime‚3731æ…9ílÐäM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652615158894592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚649264315678720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837709 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827716 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73827716 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9242615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4532157370729 % _timestamp‚1.7843664525325131e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657154266114048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144060171692032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73837709 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16779208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652615158894592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4587213972480 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34035754 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4531816820727 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652615175933952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148647404276736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4587119111168 trainer/global_step‚4299 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34035752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39801955 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4539090180096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocaÑÆ=~ted_bytes.all.freed‚657153907728384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16778938 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7536326 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049449 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657154266114048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34035752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653796115460087 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4538748833792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚3731.038735088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ÐÌ›©ï&ÑéM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚652615158894592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚649264315678720 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837709 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _step‚352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827716 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73827716 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9242615 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4532157370729 % _timestamp‚1.7843664525325131e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚657154266114048 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚144060171692032 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7536323 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73837709 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16779208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚652615158894592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4587213972480 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34035754 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4531816820727 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚652615175933952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚148647404276736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4587119111168 trainer/global_step‚4299 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34035752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39801955 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4539090180096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚657153907728384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16778938 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7536326 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚653796473049449 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚657154266114048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34035752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚653796115460087 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4538748833792 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _runtime‚3731.038735088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34035754 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0à9š;2Ò- _wandbruntime‚3731  _runtime‚3731hã½Ó _step‚353 epoch‚10 /loss/FRt2/assignment_loss‚0.893591386328141 .loss/FRt1/detection_loss‚0.580939935443905 1loss/SRqqt2/detection_loss‚0.6292774865212101 /loss/FBt1/detection_loss‚0.6442823331839967 /loss/FBt2/detection_loss‚1.2316460593913992 _runtime‚3731.039892423 2loss/FBt2/assignment_loss‚0.008398256998385853 /loss/FRt2/detection_loss‚1.4730505928295314 trainer/global_step‚4299 0loss/FRt1/assignment_loss‚1.9085862656698744 2loss/SRqqt1/assignment_loss‚0.5936367865220086 1loss/SRqqt1/detection_loss‚0.8966254168241233 2loss/SRqqt2/assignment_loss‚0.5433007583995488 1loss/FBt1/assignment_loss‚0.05825311031521218 %loss/total_loss‚9.461588388427337 % _timestamp‚1.7843664525342035e+094«,+ Ô† 2loss/SRqqt2/assignment_loss‚0.5433007583995488 1loss/FBt1/assignment_loss‚0.05825311031521218 %loss/total_loss‚9.461588388427337 % _timestamp‚1.7843664525342035e+09 _step‚353 epoch‚10 /loss/FRt2/assignment_loss‚0.893591386328141 .loss/FRt1/detection_loss‚0.580939935443905 1loss/SRqqt2/detection_loss‚0.6292774865212101 /loss/FBt1/detection_loss‚0.6442823331839967 /loss/FBt2/detection_loss‚1.2316460593913992 _runtime‚3731.039892423 2loss/FBt2/assignment_loss‚0.008398256998385853 /loss/FRt2/detection_loss‚1.4730505928295314 trainer/global_step‚4299 0loss/FRt1/assignment_loss‚1.9085862656698744 2loss/SRqqt1/assignment_loss‚0.5936367865220086 1loss/SRqqt1/detection_loss‚0.8966254168241233áQS[“Õ‚(:ˆ øŠíÒ“¿Àð gpu.0.gpu‚91.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0誰ÑÃÖ‚(:¸ øŠíҧܲó disk.sda.out‚ 82.234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 network.recv‚ 1.017442e+06 disk.rbd1.out‚9.8125 disk.sda3.in‚0 disk.rbd0.out‚0# memory_percent‚96.9801664352417 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 82.234375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 494.76953125 disk.dm-1.out‚ 82.234375* disk./.usagePercent‚39.588423445800785! disk.nvme0n1.in‚ 5004.12109375" disk.dm-0.out‚7994.23095703125' disk.nvme0c0n1.out‚7994.23095703125 cpu‚25.93023596878901 disk.dm-0.in‚ 5004.12109375 disk.rbd1.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2748.3359375 proc.cpu.threads‚17 network.sent‚ 6.829936e+06# disk.nvme0c0n1.in‚ 5004.12109375* proc.memory.percent‚16.774511337280273$ disk./.usageGB‚589.8961181640625% disk.nvme0n1.out‚7994.23095703125 disk.sda.in‚0 disk.drbd2165.in‚0·Aª:ž×‚(:“ ‡‹íÒÌçî gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Ü[¹½Ø‚(:² ‡‹íÒ°ÏÜï disk.sda2.in‚0 cpu‚25.929241251228163 disk.sda.in‚0 disk.rbd0.in‚0" disk.nvme0n1.out‚ 7999.34765625" proc.memory.rssMB‚ 2748.3359375* proc.memory.percent‚16.774511337280273) disk./.usagePercent‚39.58841627758983 disk.dm-1.out‚ 82.5234375 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 5004.12109375$ disk.nvme0c0n1.out‚ 7999.34765625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 5004.12109375 disk.dm-0.in‚ 5004.12109375 network.recv‚ 1.021972e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 7999.34765625 disk.sda2.out‚0 disk.rbd1.out‚9.8125$ memory_percent‚96.97747230529785 proc.cpu.threads‚17$ disk./.usageGB‚589.8960113525391 disk.sda3.in‚0' proc.memory.availableMB‚ 495.2109375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 82.5234375 disk.sda3.out‚ 82.5234375 network.sent‚ 6.894818e+06 disk.sda1.out‚0Fu©‘Ù‚(:† –‹íҿσí gpu.0.gpu‚39.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NRιÚ‚(:® –‹íÒ£€¬ï disk.rbd0.out‚0 disk.dm-0.in‚ 5086.50390625# disk.nvme0c0n1.in‚ 5086.50390625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.928273289398028 proc.cpu.threads‚17' proc.memory.availableMB‚ 494.7109375) disk./.usagePercent‚39.58841320549942$ disk./.usageGB‚589.8959655761719& disk.nvme0c0n1.out‚8006.9326171875 disk.sda.in‚0 disk.sda.out‚82.8125 disk.sda1.out‚0" proc.memory.rssMB‚ 2748.3359375* proc.memory.percent‚16.774511337280273! disk.nvme0n1.in‚ 5086.50390625 disk.sda2.in‚0 disk.rbd1.out‚9.875 network.recv‚ 1.025369e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0$ memory_percent‚96.98052406311035 disk.sda3.out‚82.8125 disk.drbd2165.in‚0$ disk.nvme0n1.out‚8006.9326171875! disk.dm-0.out‚8006.9326171875 network.sent‚ 6.897782e+06 disk.dm-1.out‚82.8125¯º°c2Û- _wandbruntime‚3771  _runtime‚3771çc¾tÇÜÁ & lr-AdamW/pg1‚0.0005256468503574916 & lr-AdamW/pg2‚0.0005256468503574916 trainer/global_step‚4349 % _timestamp‚1.7843664929720163e+09 _runtime‚3771.477976583 _step‚354' ×ÌÝÆ & lr-AdamW/pg1‚0.0005256468503574916 & lr-AdamW/pg2‚0.0005256468503574916 trainer/global_step‚4349 % _timestamp‚1.7843664929720163e+09 _runtime‚3771.477976583 _step‚354â3#¼2Þ- _wandbruntime‚3771  _runtime‚3771)‹Ç7à'ßÚO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4637367753728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656443321561088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401055232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661025101404251 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚659829969346560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664418759433728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656443338600448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659829986385920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40237364 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9344917 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661024743815001 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659829969346560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40247341 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40247341 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚659829986385920 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7621828 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843664929730759e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34409722 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34409720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150335567225856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚355 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145698180859904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4581422253913 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237364 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401055232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664418759433728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9344645 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145698199472128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34409722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872ÿ•èwàßO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚355 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚145698180859904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4581422253913 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40237364 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚664418401055232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚664418759433728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9344645 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚145698199472128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34409722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4637367753728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚656443321561088 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647084 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚664418401055232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚661025101404251 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚659829969346560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚664418759433728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚656443338600448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚659829986385920 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40237364 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9344917 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚661024743815001 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚659829969346560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40247341 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40247341 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚659829986385920 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7621828 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843664929730759e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34409722 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34409720 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚150335567225856 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438ã`ñ‡C2á- _wandbruntime‚3772  _runtime‚3772.«õ…ê&âäM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40256407 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664566654711808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4349 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4589444335616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7623573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4589785681920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9346947 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4582773857637 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129957 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9346684 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4582433307635 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150369913613312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 % _timestamp‚1.7843664936077244e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4638388370432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚356 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40256407 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16970257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664567013097472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656589849272320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34417354 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40246416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4638303994880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚659977210376192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40246416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150370016601088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚3772.113932767 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34417352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673761 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4589785681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16970523 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656589832232960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145731609618432 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169"o¾ºï&ãéM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74663768 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7623576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34417352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4589444335616 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7623573 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4589785681920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9346947 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4582773857637 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚661172623129957 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9346684 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4582433307635 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚150369913613312 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 % _timestamp‚1.7843664936077244e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4638388370432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚356 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40256407 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16970257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚659977227415552 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚664567013097472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚656589849272320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34417354 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40246416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4638303994880 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚659977210376192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40246416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚150370016601088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _runtime‚3772.113932767 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34417352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673761 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4589785681920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16970523 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚656589832232960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145731609618432 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40256407 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚664566654711808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4349 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40701793 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚671831506417152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚667192037867520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668401251484759 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639127210496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148038656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34791320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147369618786304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152058120508928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚3812.657652961 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4639468549632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34791322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚667192037867520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668400893895509 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17157345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚359 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34791320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40691816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75483136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7709081 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4688388241920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34791322 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632379290711 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚4399 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4688483110400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663768855154688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17157616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4632038740821 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40691816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9448535 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚671831506417152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚671831148038656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663768872194048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚667192020828160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0“÷•>å'òßO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34791322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚667192037867520 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚668400893895509 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17157345 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚359 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34791320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40691816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75483136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7709081 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4688388241920 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34791322 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4632379290711 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 trainer/global_step‚4399 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4688483110400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚663768855154688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17157616 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4632038740821 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40691816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9448535 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚671831506417152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚671831148038656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚663768872194048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚667192020828160 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4639127210496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147369637398528 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 % _timestamp‚1.7843665341515331e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚667192020828160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40701793 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚671831506417152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75493115 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚667192037867520 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚668401251484759 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4639127210496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚671831148038656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34791320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚147369618786304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152058120508928 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚3812.657652961 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4639468549632ç—ƒ8{2ó- _wandbruntime‚3813  _runtime‚3813I¹ô]&ôäM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34798952 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40700868 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚671979760080896 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40700868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147403066157056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633049794543 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40710859 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843665347875886e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40710859 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139837440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139837440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚3813.293850011 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667339278897152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4689497936896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633390344545 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667339278897152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9450614 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147403047544832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667339261857792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152092564093952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34798954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667339261857792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509813 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17161172 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75509813 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7710823 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152092456911872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4640481183744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4689409367040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34798954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStat°š£äsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7710826 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152092456911872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4640481183744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4689409367040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34798954 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7710826 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4633049794543 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40710859 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843665347875886e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40710859 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4640139837440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4640139837440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚3813.293850011 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚667339278897152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4689497936896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4633390344545 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚667339278897152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9450614 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147403047544832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚668548773210465 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚667339261857792 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152092564093952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34798954 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚667339261857792 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509813 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17161172 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75509813 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7710823è “¤2ö- _wandbruntime‚3813  _runtime‚3813m:÷‚ epoch‚10 /loss/FRt1/assignment_loss‚2.017305762160058 1loss/FBt2/assignment_loss‚0.03276263519703986 /loss/FRt1/detection_loss‚0.5800644464164022 /loss/FBt2/detection_loss‚0.6049202437911715 trainer/global_step‚4399 2loss/FBt1/assignment_loss‚0.007322482317128455 1loss/SRqqt2/detection_loss‚0.7046721613413344 % _timestamp‚1.7843665347888172e+09 _step‚361 /loss/FRt2/assignment_loss‚0.979913824653051 2loss/SRqqt1/assignment_loss‚0.6261085292483566 /loss/FRt2/detection_loss‚1.4276060877193932 %loss/total_loss‚8.973957915907327 _runtime‚3813.294795496 2loss/SRqqt2/assignment_loss‚0.5324417283906756 1loss/SRqqt1/detection_loss‚0.9993591975757575 0loss/FBt1/detection_loss‚0.46148081709695765%5yÁ ø‡ %loss/total_loss‚8.973957915907327 _runtime‚3813.294795496 2loss/SRqqt2/assignment_loss‚0.5324417283906756 1loss/SRqqt1/detection_loss‚0.9993591975757575 0loss/FBt1/detection_loss‚0.46148081709695765 epoch‚10 /loss/FRt1/assignment_loss‚2.017305762160058 1loss/FBt2/assignment_loss‚0.03276263519703986 /loss/FRt1/detection_loss‚0.5800644464164022 /loss/FBt2/detection_loss‚0.6049202437911715 trainer/global_step‚4399 2loss/FBt1/assignment_loss‚0.007322482317128455 1loss/SRqqt2/detection_loss‚0.7046721613413344 % _timestamp‚1.7843665347888172e+09 _step‚361 /loss/FRt2/assignment_loss‚0.979913824653051 2loss/SRqqt1/assignment_loss‚0.6261085292483566 /loss/FRt2/detection_loss‚1.4276060877193932é÷ÍQ”ù‚(:’ Ò‹íÒ‘÷áð gpu.0.gpu‚30.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.865' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.572999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ø[ɽú‚(:² Ò‹íÒÞПõ disk.rbd0.out‚0# disk.nvme0n1.out‚8028.576171875% disk.nvme0c0n1.out‚8028.576171875 disk.sda.out‚ 84.109375" proc.memory.rssMB‚ 2757.7734375 disk.dm-1.out‚ 84.109375 disk.drbd2165.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5168.01953125 disk.sda.in‚0 disk.sda1.out‚0) disk./.usagePercent‚39.58835304372887! disk.nvme0n1.in‚ 5168.01953125 disk.dm-0.in‚ 5168.01953125 disk.rbd1.in‚0' proc.memory.availableMB‚ 485.8671875) proc.memory.percent‚16.83211326599121 network.sent‚ 6.974757e+06 network.recv‚ 1.039062e+06$ disk./.usageGB‚589.8950691223145 disk.sda2.in‚0$ memory_percent‚97.03450202941895 disk.dm-0.out‚8028.576171875 disk.rbd1.out‚10.0625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 84.109375 cpu‚25.936646097843443 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0‘:2û- _wandbruntime‚3830  _runtime‚3830’ ÖÃÒ=üÌ{ =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7018525004386902 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4730859100818634 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7386715412139893 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6647108197212219 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7474772334098816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845777153968811 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28399938344955444 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.712090790271759 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9872215986251831 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8645373582839966 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896196722984314 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04242892935872078 *jet/accuracy_1_of_2‚0.8016457557678223 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6785958409309387 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7507297992706299 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7530652284622192 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_5_of_5‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.652332603931427 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955244660377502 particle/accuracy_1_of_3‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.544243574142456 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7199172973632812 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31644368171691895 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7919973731040955 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4399 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20687705278396606 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9792395234107971 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20463444292545319 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13446830213069916 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6419268846511841 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9215726256370544 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22250495851039886 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03866887465119362 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967522621154785 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967522621154785 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4749835729598999 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8519275188446045 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7462195158004761 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969159960746765 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934069514274597 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.945494713494586 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9251583814620972 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6595673561096191 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21585048735141754 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.660998523235321 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7894369959831238 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6369553804397583 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6555252075195312 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8479024767875671 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945777654647827 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19243557751178741 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0712842345237732 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936597943305969 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24058210849761963 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47275465726852417 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1983262300491333 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44573497772216797 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6061774492263794 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_3_of_4‚0.9942177534103394 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8893227577209473 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.993106484413147 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6647108197212219 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 /particle/accuracy_4_of_4‚0.9328165650367737 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23344285786151886 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7038604617118835 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5393291711807251 +particle/sensitivity‚0.6227551698684692 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22250495851039886 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8024399876594543 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873001575469971 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7547114491462708 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04257999733090401 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22204603254795074 +jet/accuracy_1_of_1‚0.33544793725013733 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20438861846923828 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.593338131904602 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0713922381401062 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1921866089105606 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7068471312522888 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1506940871477127 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19467878341674805 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962116479873657 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2341025024652481 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.994591236114502 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21479740738868713 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39608991146087646 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19050662219524384 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.609375 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16994129121303558 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23343347012996674 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_3‚0.9258400797843933 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7939465641975403 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7541654706001282 % _timestamp‚1.7843665520067303e+09 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1802103966474533 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8468632698059082 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39290136098861694 *jet/accuracy_1_of_4‚0.9679604768753052 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8729625940322876 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04242892935872078 _runtime‚3830.513355006 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02955423854291439 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8369794487953186 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7254346013069153 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46451810002326965 *jet/accuracy_2_of_3‚0.5644054412841797 particle/accuracy_2_of_5‚1 epoch‚10 _step‚362 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7179847359657288 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6177619099617004 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6621897220611572 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6623051762580872 7validation_average_jet_accuracy‚0.37808236479759216 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6025711297988892 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9851977825164795 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_3_of_3‚0.0724007859826088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6641223430633545 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.756092369556427 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06227647140622139 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4330669939517975 *jet/accuracy_2_of_2‚0.1537865847349167 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2224700152873993 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8369794487953186 +particle/specificity‚0.9090413451194763 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6855619549751282 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +jet/accuracy_3_of_4‚0.09115542471408844 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8568406105041504 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7443511486053467 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12122140824794769 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7529136538505554 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23463904857635498 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5974487662315369 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06218221038579941 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2879205048084259 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7039943933486938 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05037920922040939 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13691236078739166 'particle/accuracy‚0.847567081451416 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.641555666923523 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05630459636449814 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32433441281318665 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 /particle/accuracy_3_of_3‚0.9994933605194092 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976010918617249 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02249600924551487 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0e x™ýÑ{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 +jet/accuracy_3_of_4‚0.09115542471408844 particle/accuracy_2_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8568406105041504 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_2‚1 jet/accuracy_4_of_5‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9854875206947327 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7443511486053467 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1275000125169754 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12122140824794769 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7529136538505554 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23463904857635498 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5974487662315369 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06218221038579941 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2879205048084259 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7039943933486938 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05037920922040939 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13691236078739166 'particle/accuracy‚0.847567081451416 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.641555666923523 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05630459636449814 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.32433441281318665 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 /particle/accuracy_3_of_3‚0.9994933605194092 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976010918617249 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02249600924551487 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7018525004386902 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4730859100818634 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7386715412139893 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6647108197212219 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7474772334098816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845777153968811 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28399938344955444 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.712090790271759 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9872215986251831 ("Purity/1FRt2SRøýû2>/qqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8645373582839966 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9896196722984314 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04242892935872078 *jet/accuracy_1_of_2‚0.8016457557678223 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6785958409309387 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7507297992706299 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7530652284622192 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 jet/accuracy_5_of_5‚0 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.652332603931427 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955244660377502 particle/accuracy_1_of_3‚1 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.544243574142456 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7199172973632812 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31644368171691895 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7919973731040955 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4399 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20687705278396606 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9792395234107971 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20463444292545319 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13446830213069916 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6419268846511841 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9215726256370544 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22250495851039886 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03866887465119362 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9967522621154785 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9967522621154785 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4749835729598999 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8519275188446045 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7462195158004761 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969159960746765 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934069514274597 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942795634269714 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚8.945494713494586 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9251583814620972 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6595673561096191 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21585048735141754 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.660998523235321 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7894369959831238 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6369553804397583 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6555252075195312 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8479024767875671 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945777654647827 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19243557751178741 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0712842345237732 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9936597943305969 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9877755045890808 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24058210849761963 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47275465726852417 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1983262300491333 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44573497772216797 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6061774492263794 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_3_of_4‚0.9942177534103394 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8893227577209473 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.993106484413147 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6647108197212219 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 /particle/accuracy_4_of_4‚0.9328165650367737 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23344285786151886 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7038604617118835 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5393291711807251 +particle/sensitivity‚0.6227551698684692 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22250495851039886 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.8024399876594543 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9873001575469971 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7547114491462708 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.534583330154419 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04257999733090401 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22204603254795074 +jet/accuracy_1_of_1‚0.33544793725013733 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20438861846923828 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.593338131904602 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0713922381401062 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1921866089105606 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7068471312522888 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1506940871477127 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19467878341674805 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962116479873657 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2341025024652481 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.994591236114502 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21479740738868713 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39608991146087646 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19050662219524384 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.609375 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16994129121303558 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23343347012996674 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 *jet/accuracy_1_of_3‚0.9258400797843933 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7939465641975403 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7541654706001282 % _timestamp‚1.7843665520067303e+09 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1802103966474533 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8468632698059082 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39290136098861694 *jet/accuracy_1_of_4‚0.9679604768753052 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8729625940322876 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04242892935872078 _runtime‚3830.513355006 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_3_of_5‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02955423854291439 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8369794487953186 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7254346013069153 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46451810002326965 *jet/accuracy_2_of_3‚0.5644054412841797 particle/accuracy_2_of_5‚1 epoch‚10 _step‚362 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7179847359657288 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6177619099617004 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6621897220611572 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6623051762580872 7validation_average_jet_accuracy‚0.37808236479759216 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6025711297988892 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9851977825164795 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_3_of_3‚0.0724007859826088 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6641223430633545 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.756092369556427 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06227647140622139 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4330669939517975 *jet/accuracy_2_of_2‚0.1537865847349167 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2224700152873993 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8369794487953186 +particle/specificity‚0.9090413451194763 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6855619549751282ê_åì“þ‚(:ˆ á‹íÒÃçÖï gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚173.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.6884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓjÛÀÿ‚(:µ á‹íÒ›óÝñ! disk.nvme0n1.in‚ 5168.01953125$ disk./.usageGB‚589.8959693908691 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚25.940434391715044 disk.sda3.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2767.35546875 network.sent‚ 7.050513e+06) disk./.usagePercent‚39.58841346150695 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 5168.01953125$ memory_percent‚95.46492099761963 proc.cpu.threads‚17 disk.drbd2165.out‚0# disk.nvme0n1.out‚8277.478515625 disk.sda2.in‚0 disk.rbd1.out‚10.0625 disk.dm-1.out‚ 84.421875 network.recv‚ 1.043656e+06* proc.memory.percent‚16.890597343444824 disk.sda.out‚ 84.421875 disk.rbd0.in‚0 disk.dm-0.out‚8277.478515625% disk.nvme0c0n1.out‚8277.478515625( proc.memory.availableMB‚ 743.02734375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 5168.01953125 disk.sda3.out‚ 84.421875 disk.drbd2165.in‚0'ÄÌž€‚(:“ ð‹íÒ£±í gpu.0.gpu‚76.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î§ýrÈ‚(:½ ð‹íÒñ™Òï disk.sda.out‚ 84.6796875 disk.dm-1.in‚0* proc.memory.percent‚16.890597343444824 network.sent‚ 7.096948e+06$ disk./.usageGB‚589.8957138061523 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0! disk.nvme0n1.in‚ 5168.01953125" disk.dm-0.out‚8528.41552734375 disk.sda1.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚39.588396309002164 disk.sda1.in‚0 disk.sda3.out‚ 84.6796875# proc.memory.rssMB‚ 2767.35546875# disk.nvme0c0n1.in‚ 5168.01953125' disk.nvme0c0n1.out‚8528.41552734375( proc.memory.availableMB‚ 742.82421875% disk.nvme0n1.out‚8528.41552734375 disk.sda2.in‚0 disk.rbd1.out‚10.125 disk.sda3.in‚0 disk.dm-1.out‚ 84.6796875 cpu‚25.93953401842064 disk.sda2.out‚0$ memory_percent‚95.46616077423096 disk.rbd0.in‚0 disk.rbd1.in‚0 network.recv‚ 1.047828e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 5168.01953125MwÓŸž‚‚(:“ ÿ‹íÒ”¢àí gpu.0.gpu‚87.0 gpu.0.memory‚23+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚250.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø¯Èƒ‚(:½ ÿ‹íÒàý–ï disk.rbd1.in‚0 disk.sda.out‚ 84.9921875 disk.rbd0.in‚0% disk.nvme0n1.out‚8531.77001953125* proc.memory.percent‚16.890597343444824 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.051453e+06 disk.rbd1.out‚10.125 disk.dm-1.in‚0 disk.dm-0.in‚ 5168.01953125 network.sent‚ 7.09987e+06* disk./.usagePercent‚39.588396309002164' disk.nvme0c0n1.out‚8531.77001953125 disk.dm-1.out‚ 84.9921875 cpu‚25.938553759796253 disk.sda.in‚0 disk.sda3.out‚ 84.9921875$ memory_percent‚95.46654224395752 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5168.01953125 disk.sda3.in‚0! disk.nvme0n1.in‚ 5168.01953125" disk.dm-0.out‚8531.77001953125# proc.memory.rssMB‚ 2767.35546875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 742.76171875$ disk./.usageGB‚589.8957138061523Ñ9“2„- _wandbruntime‚3872  _runtime‚3872¹,ºÆ…À _runtime‚3872.941160463 _step‚363 & lr-AdamW/pg1‚0.0005228576015917113 & lr-AdamW/pg2‚0.0005228576015917113 trainer/global_step‚4449 $ _timestamp‚1.784366594435211e+09êÄV ˆŠ& lr-AdamW/pg1‚0.0005228576015917113 & lr-AdamW/pg2‚0.0005228576015917113 trainer/global_step‚4449 $ _timestamp‚1.784366594435211e+09 _runtime‚3872.941160463 _step‚363ë¥i Ù2‡- _wandbruntime‚3872  _runtime‚3872‡š€ß'ˆÙO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681191553259008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153814800712704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4757733099520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4700374952963 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17381367 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4707897259520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41226572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149056987424768 _runtime‚3872.942377186 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673003069001728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681191194880512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4757813287936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35293588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41226572 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41216595 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7798848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194880512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502853 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677703784504581 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677703426915331 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676483655999488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7798851 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4707555920384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9582519 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚676483655999488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676483638960128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153814701912064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚676483638960128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17381614 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41216595 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149056968812544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ØBt‰ÞO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35293586 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673003051962368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843665944362695e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4707897259520 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚676483655999488 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7798851 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4707555920384 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚76520160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9582519 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚676483655999488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚676483638960128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153814701912064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚676483638960128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_t~æÖprain_batch_start/num_oom_rejections‚0 trainer/global_step‚4449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17381614 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41216595 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149056968812544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚681191553259008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153814800712704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4757733099520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4700374952963 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17381367 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4707897259520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41226572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚149056987424768 _runtime‚3872.942377186 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚673003069001728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚681191194880512 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4757813287936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35293588 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41226572 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41216595 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7798848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚681191194880512 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4700715502853 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚677703784504581 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚677703426915331 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35301218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚3873.575710506 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673149562634240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676630879989760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41225647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚676630897029120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4708909893632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚4449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4758760606208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526865 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9584845 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17385182 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7800596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41225647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149090416183296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677851306230287 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4708909893632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673149579673600 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚676630897029120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7800593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843665950695055e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚677851306230287 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4708909893632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673149579673600 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚676630897029120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7800593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 % _timestamp‚1.7843665950695055e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35301218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _runtime‚3873.575710506 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673149562634240 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚676630879989760 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41225647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚676630897029120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4708909893632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 trainer/global_step‚4449 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4758760606208 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526865 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9584845 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17385182 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7800596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41225647 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376í Wæ¯2- _wandbruntime‚3873  _runtime‚3873‚æ[Ž€ 2loss/SRqqt2/assignment_loss‚0.4788750072366081 1loss/FBt1/assignment_loss‚0.04791465614159665 1loss/SRqqt2/detection_loss‚0.6342172021631786 /loss/FBt1/detection_loss‚0.6028122998753805 _step‚366 /loss/FRt1/detection_loss‚0.5553519140749839 /loss/FRt2/detection_loss‚1.4324321919437375 %loss/total_loss‚9.018438754205842 0loss/FRt1/assignment_loss‚2.0472637017770072 0loss/FRt2/assignment_loss‚0.8124435262378996 .loss/FBt2/detection_loss‚0.598932712831918 trainer/global_step‚4449 1loss/FBt2/assignment_loss‚0.18897709147677566 0loss/SRqqt1/detection_loss‚1.032301301375338 epoch‚11 % _timestamp‚1.7843665950707917e+09 _runtime‚3873.576523244 2loss/SRqqt1/assignment_loss‚0.5869171490714185×Iw … 0loss/FRt1/assignment_loss‚2.0472637017770072 0loss/FRt2/assignment_loss‚0.8124435262378996 .loss/FBt2/detection_loss‚0.598932712831918 trainer/global_step‚4449 1loss/FBt2/assignment_loss‚0.18897709147677566 0loss/SRqqt1/detection_loss‚1.032301301375338 epoch‚11 % _timestamp‚1.7843665950707917e+09 _runtime‚3873.576523244 2loss/SRqqt1/assignment_loss‚0.5869171490714185 2loss/SRqqt2/assignment_loss‚0.4788750072366081 1loss/FBt1/assignment_loss‚0.04791465614159665 1loss/SRqqt2/detection_loss‚0.6342172021631786 /loss/FBt1/detection_loss‚0.6028122998753805 _step‚366 /loss/FRt1/detection_loss‚0.5553519140749839 /loss/FRt2/detection_loss‚1.4324321919437375 %loss/total_loss‚9.018438754205842îLx2Úž‚(:“ ŽŒíҟƯî gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚242.388' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.477599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ö ŽÃ‘‚(:¸ ŽŒí҃ɦï disk.sda2.out‚0 network.recv‚ 1.054699e+06 cpu‚25.937572900101095% disk.nvme0n1.out‚8536.01318359375* proc.memory.percent‚16.890597343444824 network.sent‚ 7.104962e+06 disk.sda3.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚95.46966552734375$ disk./.usageGB‚589.8953895568848 disk.dm-1.out‚ 85.2265625 disk.dm-0.in‚ 5214.41015625" disk.dm-0.out‚8536.01318359375# disk.nvme0c0n1.in‚ 5214.41015625' disk.nvme0c0n1.out‚8536.01318359375 disk.sda.in‚0 disk.sda.out‚ 85.2265625) disk./.usagePercent‚39.58837454836175 disk.sda3.out‚ 85.2265625 disk.sda1.in‚0" proc.memory.availableMB‚742.25# proc.memory.rssMB‚ 2767.35546875 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 5214.41015625 disk.rbd1.out‚10.1875 proc.cpu.threads‚17ª?Í¿ž’‚(:“ ŒíÒ ¼ôí gpu.0.gpu‚74.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kõ^(¸“‚(:­ ŒíÒûå³ð disk.dm-1.out‚85.46875 disk.rbd1.out‚10.1875% disk.nvme0c0n1.out‚8539.205078125 disk.dm-0.in‚ 5214.41015625 disk.sda3.in‚0* proc.memory.percent‚16.890621185302734) disk./.usagePercent‚39.58837096425627 disk.drbd2165.out‚0 disk.rbd0.in‚0! disk.nvme0n1.in‚ 5214.41015625 disk.sda.in‚0 disk.sda.out‚85.46875 network.sent‚ 7.169853e+06# disk./.usageGB‚589.895336151123 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 5214.41015625 disk.sda3.out‚85.46875 network.recv‚ 1.059131e+06 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 742.734375! proc.memory.rssMB‚ 2767.359375 proc.cpu.threads‚17# disk.nvme0n1.out‚8539.205078125 disk.rbd1.in‚0 disk.sda1.out‚0$ memory_percent‚95.46670913696289 disk.drbd2165.in‚0 disk.dm-0.out‚8539.205078125 disk.sda1.in‚0 cpu‚25.936777191650826Ëek¯2”- _wandbruntime‚3913  _runtime‚3913@™ùïÇ•Á & lr-AdamW/pg1‚0.0005214405956045347 & lr-AdamW/pg2‚0.0005214405956045347 trainer/global_step‚4499 % _timestamp‚1.7843666353612936e+09 _runtime‚3913.867271137 _step‚367 0:Ì–Æ & lr-AdamW/pg2‚0.0005214405956045347 trainer/global_step‚4499 % _timestamp‚1.7843666353612936e+09 _runtime‚3913.867271137 _step‚367 & lr-AdamW/pg1‚0.0005214405956045347ï¿òå2—- _wandbruntime‚3913  _runtime‚3913Ó^c$ ˜ÚO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4758592761344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41671047 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3913.871778461 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7886098 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4809096564224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685079934585089 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17572463 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77356212 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚4499 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941863936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9686106 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚685079576995839 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 % _timestamp‚1.7843666353623917e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35675186 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683845707481088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚680328585555968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4751331989761 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4750991439871 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41681024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155537521915392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688604300242432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154¬VšÒå'™ßO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155537418920448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚680328602595328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9686362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4809012181504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35675188 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚683845707481088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚4499 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688603941863936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9686106 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚685079576995839 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35675188 % _timestamp‚1.7843666353623917e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35675186 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚683845707481088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚680328585555968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4751331989761 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4750991439871 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41681024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155537521915392 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 _step‚368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688604300242432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4758592761344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41671047 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚84382720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3913.871778461 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7886098 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4809096564224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚685079934585089 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17572463 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77356212 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17575969 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4752343043595 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843666361811008e+09 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688752553906176 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9688375 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9688126 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41680099 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683992931471360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35682820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41690090 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚685227456310795 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4810124993536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688752195520512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683992931471360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4759264049152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155571979103232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155571876115456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚680475096227840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17576221 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7887846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4810040617984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚680475113267200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41680099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35682818 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4759605395456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150761835497472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35682820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4759605395456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚4499 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚688752553906176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4752002493593 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150761854109696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17575969 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4752343043595 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 % _timestamp‚1.7843666361811008e+09 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚688752553906176 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9688375 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9688126 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41680099 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚683992948510720 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚683992931471360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35682820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41690090 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 WCDeviceStatsMonitor.on_‘cøÖÏtrain_batch_end/requested_bytes.all.allocated‚685227456310795 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7887843 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4810124993536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688752195520512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚683992931471360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4759264049152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155571979103232 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155571876115456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚680475096227840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17576221 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7887846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4810040617984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚680475113267200 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41680099 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35682818 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4759605395456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150761835497472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2ñöôþ2- _wandbruntime‚3914  _runtime‚3914VW©® ž„ 0loss/FRt1/assignment_loss‚2.0321173114436015 3loss/SRqqt2/assignment_loss‚0.48721985161067016 0loss/SRqqt1/detection_loss‚0.916732722552854 /loss/FBt1/detection_loss‚0.2836992677107143 /loss/FBt2/detection_loss‚0.9198813002373352 3loss/FBt2/assignment_loss‚0.0015871315544331555 % _timestamp‚1.7843666361824436e+09 /loss/FRt1/detection_loss‚0.5455685775979042 /loss/FRt2/detection_loss‚1.3401416424140162 trainer/global_step‚4499 _runtime‚ 3914.68822344 0loss/FRt2/assignment_loss‚0.8607774064128781 2loss/SRqqt1/assignment_loss‚0.5872332996014333 2loss/FBt1/assignment_loss‚0.018073678710703808 1loss/SRqqt2/detection_loss‚0.6250132999089207 %loss/total_loss‚8.618045489755463 epoch‚11 _step‚370öÀ🉠0loss/FRt2/assignment_loss‚0.8607774064128781 2loss/SRqqt1/assignment_loss‚0.5872332996014333 2loss/FBt1/assignment_loss‚0.018073678710703808 1loss/SRqqt2/detection_loss‚0.6250132999089207 %loss/total_loss‚8.618045489755463 epoch‚11 _step‚370 0loss/FRt1/assignment_loss‚2.0321173114436015 3loss/SRqqt2/assignment_loss‚0.48721985161067016 0loss/SRqqt1/detection_loss‚0.916732722552854 /loss/FBt1/detection_loss‚0.2836992677107143 /loss/FBt2/detection_loss‚0.9198813002373352 3loss/FBt2/assignment_loss‚0.0015871315544331555 % _timestamp‚1.7843666361824436e+09 /loss/FRt1/detection_loss‚0.5455685775979042 /loss/FRt2/detection_loss‚1.3401416424140162 trainer/global_step‚4499 _runtime‚ 3914.68822344òñ)uHž ‚(:“ ¬ŒíÒ¦¥çí gpu.0.gpu‚42.0 gpu.0.memory‚14+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06l VÇ¡‚(:¼ ¬ŒíÒ€éÓñ disk.dm-1.out‚ 85.79296875 disk.dm-0.in‚ 5214.41015625' disk.nvme0c0n1.out‚8544.48681640625 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 network.recv‚ 1.062484e+06$ disk./.usageGB‚589.8952331542969 disk.drbd2165.out‚0& proc.memory.availableMB‚ 742.203125 disk.drbd2165.in‚0 disk.rbd0.in‚0! disk.nvme0n1.in‚ 5214.41015625 disk.sda.out‚ 85.79296875 disk.sda1.out‚0 disk.sda3.in‚0 cpu‚25.935858542261634 proc.cpu.threads‚17% disk.nvme0n1.out‚8544.48681640625 disk.rbd1.out‚10.25$ memory_percent‚95.46995162963867 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 5214.41015625 disk.sda3.out‚ 85.79296875! proc.memory.rssMB‚ 2767.359375 disk.dm-1.in‚0" disk.dm-0.out‚8544.48681640625 disk.sda2.out‚0* proc.memory.percent‚16.890621185302734 network.sent‚ 7.185816e+06* disk./.usagePercent‚39.588364052052846 zž¢‚(:“ »ŒíÒöé„í gpu.0.gpu‚88.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚262.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.468399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G,$ꣂ(:· »ŒíÒïîøî$ disk./.usageGB‚589.8960380554199 disk.sda1.out‚0 disk.dm-1.out‚ 86.10546875 disk.drbd2165.out‚0 disk.dm-0.in‚ 5214.41015625$ memory_percent‚95.46864032745361( proc.memory.availableMB‚ 742.41796875 disk.sda.out‚ 86.10546875 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.890621185302734! disk.nvme0n1.in‚ 5214.41015625 disk.dm-0.out‚8548.583984375 disk.rbd1.out‚10.25 cpu‚25.93487537506654 proc.cpu.threads‚17 network.recv‚ 1.067209e+06# disk.nvme0c0n1.in‚ 5214.41015625 disk.rbd0.out‚0 disk.sda2.out‚0! proc.memory.rssMB‚ 2767.359375 disk.sda.in‚0* disk./.usagePercent‚39.588418069642564 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 86.10546875# disk.nvme0n1.out‚8548.583984375 network.sent‚ 7.235791e+06% disk.nvme0c0n1.out‚8548.583984375u¸¯Ü“¤‚(:ˆ ÊŒíÒĶ¬í gpu.0.gpu‚87.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚237.556' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.5112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùà}߸¥‚(:­ ÊŒíÒ§Ïï disk.drbd2165.out‚0! disk.dm-0.out‚8556.2685546875 disk.nvme0n1.in‚ 5270.6171875 disk.sda.in‚0 disk.sda3.out‚86.34375 disk.dm-1.in‚0 disk.dm-1.out‚86.34375 disk.dm-0.in‚ 5270.6171875 disk.rbd1.out‚10.375 disk.drbd2165.in‚0' proc.memory.availableMB‚ 742.1640625 proc.cpu.threads‚17 network.recv‚ 1.070409e+06$ disk./.usageGB‚589.8949317932129 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda.out‚86.34375 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚95.47019004821777! proc.memory.rssMB‚ 2767.359375 cpu‚25.93391488715316 network.sent‚ 7.240738e+06 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 5270.6171875) disk./.usagePercent‚39.58834382745764 disk.rbd0.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.890621185302734$ disk.nvme0n1.out‚8556.2685546875 disk.sda2.in‚0& disk.nvme0c0n1.out‚8556.2685546875å­®2¦- _wandbruntime‚3954  _runtime‚3954ôë©ÄƧÀ & lr-AdamW/pg2‚0.0005200087895778881 trainer/global_step‚4549 $ _timestamp‚1.784366676479871e+09 _runtime‚3954.985796137 _step‚371 & lr-AdamW/pg1‚0.0005200087895778881vC…ˆË¨Å $ _timestamp‚1.784366676479871e+09 _runtime‚3954.985796137 _step‚371 & lr-AdamW/pg1‚0.0005200087895778881 & lr-AdamW/pg2‚0.0005200087895778881 trainer/global_step‚4549ó#ª2©- _wandbruntime‚3954  _runtime‚3954…Ñà'ªÚO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 _runtime‚3954.987059683 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚78192264 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42135476 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7973351 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36056788 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4801607926779 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4860289725952 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚4549 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚691207758962688 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚692455727076347 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚692456084665597 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17763208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42125499 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4808946924032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36056786 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4809288263168 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚687654119149568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚691207741923328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688847360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696017047225856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚696017047225856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42125499 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4860199051776 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7973348 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152399863277568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843666764809594e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4809288263168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4801948476669 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696016688847360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9789593 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157260153003520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157260043717120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚687654136188928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42135476 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78182285 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4808946924032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17762941 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚696016688847360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚696017047225856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚696017047225856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42125499 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4860199051776 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7973348 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152399863277568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843666764809594e+09 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 _step‚372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚691207758962688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4809288263168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4801948476669 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚696016688847360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9789593 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157260153003520 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157260043717120 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚687654136188928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42135476 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78182285 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4808946924032 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17762941 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843666771146455e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300889600 NADeviceStatsMonitor.@¯)~ on_train_batch_end/allocation.large_pool.freed‚36064418 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7975096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530503 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4802618980501 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36064420 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚3955.620803761 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696165300889600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42144542 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9791629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157294599910400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152433273423872 _step‚373 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17767002 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚687800629821440 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78208962 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691354999992320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300897280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4809959550976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959550976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134551 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17766722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692603248801941 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4861307874304 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0áÄÖ ï&®éM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚696165300889600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42144542 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42134551 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9791629 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157294599910400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152433273423872 _step‚373 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17767002 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚687800629821440 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78208962 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚691354999992320 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4810300897280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4809959550976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7975093 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4809959550976 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42134551 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17766722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78198969 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚692603248801941 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4861307874304 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚691354999992320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152433292036096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843666771146455e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚696165300889600 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36064418 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7975096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4802959530503 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4802618980501 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36064420 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚3955.620803761 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚691354982952960 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888õ8äq-2¯- _wandbruntime‚3955  _runtime‚3955By*¯ °… _runtime‚3955.621974231 _step‚374 2loss/FBt1/assignment_loss‚0.023930817434442395 1loss/SRqqt2/detection_loss‚0.6914474837317625 /loss/FBt2/detection_loss‚0.9683475709062511 %loss/total_loss‚9.074437185288579 epoch‚11 0loss/FRt1/assignment_loss‚2.0014521140337735 0loss/FRt2/assignment_loss‚0.9953662659440722 /loss/FRt2/detection_loss‚1.4172890335744324 1loss/SRqqt1/detection_loss‚0.8529047047143633 % _timestamp‚1.7843666771162884e+09 2loss/SRqqt1/assignment_loss‚0.6362581932273778 2loss/SRqqt2/assignment_loss‚0.5079823276103484 3loss/FBt2/assignment_loss‚0.0026837773935133764 /loss/FRt1/detection_loss‚0.5952660598036511 /loss/FBt1/detection_loss‚0.3815088369145916 trainer/global_step‚4549²I ±Š 2loss/SRqqt1/assignment_loss‚0.6362581932273778 2loss/SRqqt2/assignment_loss‚0.5079823276103484 3loss/FBt2/assignment_loss‚0.0026837773935133764 /loss/FRt1/detection_loss‚0.5952660598036511 /loss/FBt1/detection_loss‚0.3815088369145916 trainer/global_step‚4549 _runtime‚3955.621974231 _step‚374 2loss/FBt1/assignment_loss‚0.023930817434442395 1loss/SRqqt2/detection_loss‚0.6914474837317625 /loss/FBt2/detection_loss‚0.9683475709062511 %loss/total_loss‚9.074437185288579 epoch‚11 0loss/FRt1/assignment_loss‚2.0014521140337735 0loss/FRt2/assignment_loss‚0.9953662659440722 /loss/FRt2/detection_loss‚1.4172890335744324 1loss/SRqqt1/detection_loss‚0.8529047047143633 % _timestamp‚1.7843666771162884e+09öªÚ$³ž²‚(:“ ÙŒíÒžÇþì gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚114.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.839199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,v—V¼³‚(:± ÙŒíÒÏ èð disk.sda3.in‚0 disk.sda3.out‚ 86.76171875 disk.dm-1.in‚0# disk.nvme0n1.out‚8561.318359375 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.93370827752255 network.sent‚ 7.245572e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 5270.6171875 disk.dm-0.out‚8561.318359375 disk.sda2.in‚0! proc.memory.rssMB‚ 2767.359375$ disk./.usageGB‚589.8950614929199 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚ 86.76171875" disk.nvme0c0n1.in‚ 5270.6171875 proc.cpu.threads‚17 disk.drbd2165.out‚0# memory_percent‚96.9728946685791* proc.memory.percent‚16.890621185302734 disk.rbd0.out‚0 disk.dm-0.in‚ 5270.6171875 disk.rbd1.out‚10.375 disk.dm-1.out‚ 86.76171875% disk.nvme0c0n1.out‚8561.318359375' proc.memory.availableMB‚ 495.9609375 network.recv‚ 1.073825e+06( disk./.usagePercent‚39.5883525317138‡°6‘´‚(:† èŒíÒ…ÅÀí gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a¹;ŵ‚(:º èŒíÒê¯úï% disk.nvme0n1.out‚8567.58349609375 disk.drbd2165.out‚0$ memory_percent‚96.96946144104004 disk.nvme0n1.in‚ 5270.6171875 disk.sda2.in‚0 disk.rbd1.out‚10.4375 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 5270.6171875 disk.sda.in‚0) disk./.usagePercent‚39.58835457977408# disk./.usageGB‚589.895092010498 disk.sda.out‚ 87.17578125 disk.dm-1.in‚0 disk.drbd2165.in‚0! proc.memory.rssMB‚ 2767.359375* proc.memory.percent‚16.890621185302734 proc.cpu.threads‚17' proc.memory.availableMB‚ 496.5234375 network.sent‚ 7.308466e+06 disk.sda2.out‚0' disk.nvme0c0n1.out‚8567.58349609375 disk.dm-0.in‚ 5270.6171875 disk.rbd1.in‚0 disk.sda1.in‚0" disk.dm-0.out‚8567.58349609375 disk.sda3.out‚ 87.17578125 disk.dm-1.out‚ 87.17578125 cpu‚25.935593556766083 network.recv‚ 1.078254e+06 disk.rbd0.out‚0Õ;e퓶‚(:ˆ ÷ŒíÒøÂÁí gpu.0.gpu‚94.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚110.952' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1904 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’îìÀ·‚(:µ ÷ŒíÒéÞàï( proc.memory.availableMB‚ 496.26953125 proc.cpu.threads‚17 network.recv‚ 1.081868e+06" disk.nvme0c0n1.in‚ 5270.6171875 disk.drbd2165.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚8570.8056640625 disk.dm-0.in‚ 5270.6171875 network.sent‚ 7.313505e+06 disk.rbd0.out‚0 disk.rbd1.out‚10.4375* proc.memory.percent‚16.890621185302734 disk.sda3.in‚0 disk.sda1.in‚0$ disk./.usageGB‚589.8956527709961 disk.sda.out‚ 87.6171875! disk.dm-0.out‚8570.8056640625 disk.sda2.in‚0& disk.nvme0c0n1.out‚8570.8056640625 disk.sda3.out‚ 87.6171875 disk.nvme0n1.in‚ 5270.6171875 disk.sda2.out‚0 cpu‚25.93695776080937 disk.sda.in‚0 disk.dm-1.out‚ 87.6171875 disk.drbd2165.out‚0 disk.rbd1.in‚0# memory_percent‚96.9710111618042! proc.memory.rssMB‚ 2767.359375* disk./.usagePercent‚39.588392212881615 disk.sda1.out‚0 disk.dm-1.in‚0§º2¸- _wandbruntime‚3996  _runtime‚3996†UǹÁ & lr-AdamW/pg2‚0.0005185622754739967 trainer/global_step‚4599 % _timestamp‚1.7843667182217999e+09 _runtime‚3996.727795236 _step‚375 & lr-AdamW/pg1‚0.00051856227547399673ͪoÌºÆ _step‚375 & lr-AdamW/pg1‚0.0005185622754739967 & lr-AdamW/pg2‚0.0005185622754739967 trainer/global_step‚4599 % _timestamp‚1.7843667182217999e+09 _runtime‚3996.727795236÷{Ð#H2»- _wandbruntime‚3996  _runtime‚3996/C3ܼÚO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42589928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17953542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17953280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9892941 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _step‚376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703429794209280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8060598 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158982543185408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154071282591744 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154071301203968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚698569810444288 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158982648277504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4852224413687 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703429435830784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚698569810444288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4859983764992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843667182228615e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859642425856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859642425856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4852564963577 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/a{øctive.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435830784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚703429794209280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694979669782528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694979652743168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9892682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3996.732754761 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4911347073536 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36438388c>‡å'½ßO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚154071301203968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚698569810444288 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚158982648277504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4852224413687 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚703429435830784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚698569810444288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4859983764992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7843667182228615e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4859642425856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4859642425856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4852564963577 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚703429435830784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚703429794209280 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚694979669782528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚694979652743168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9892682 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3996.732754761 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4911347073536 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36438388 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42589928 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚698569793404928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17953542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17953280 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42589928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9892941 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 _step‚376 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚703429794209280 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8060598 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158982543185408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚154071282591744 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269ø7Bè…2¾- _wandbruntime‚3997  _runtime‚3997°„¹–ê&¿äM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598994 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17957284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703578047873024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36446018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882449 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚377 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860655052800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698717034434560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42589003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154104711350272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36446020 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42589003 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42598994 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4912370743296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703577689487360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698717051473920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695126163415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4912275881984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚699979756471811 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17956999 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159016987232256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695126180454400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843667188552315e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9894938 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655052800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062343 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚2691oU>ï&ÀéM LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4853235467409 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚698717034434560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4912370743296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚703577689487360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚698717051473920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚695126163415040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4912275881984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚699979756471811 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17956999 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159016987232256 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚695126180454400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843667188552315e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚282 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9894938 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4860655052800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062343 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42598994 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17957284 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚703578047873024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36446018 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚699979398882449 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _step‚377 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4860655052800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚698717034434560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚285 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42589003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154104711350272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36446018 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36446020 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42589003 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42598994 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4910337927680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4962510223360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843667597205732e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4902840900595 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4910679266816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36819986 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182814208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚705931844886528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4910679266816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710842541192704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864368 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155742739130368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155742720518144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8147851 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36819988 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182814208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43034403 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8147848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160705249353728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43034403 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4962421646336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4910337927680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚380 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18144441 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚4038.226906042 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160705142164480 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 óÙúå'ÏßO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4910337927680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4962510223360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843667597205732e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4902840900595 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4910679266816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚4649 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36819986 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710842182814208 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚705931844886528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4910679266816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710842541192704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79864368 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155742739130368 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155742720518144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8147851 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36819988 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710842182814208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43034403 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8147848 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚160705249353728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43034403 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4962421646336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4910337927680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚380 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18144441 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _runtime‚4038.226906042 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚160705142164480 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79854389 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4911691900928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710990794856448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36827618 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚707355906552319 trainer/global_step‚4649 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4904192504319 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155776167888896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9998699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43043455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4911350554624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8149596 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160739583163904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36827620 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚4038.861813568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706079085916160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36827620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706079085916160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707355548962957 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18148024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702451697008640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0%Œ!ßï&ÒéM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18148024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚702451697008640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881066 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36827618 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4911691900928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79871073 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710990794856448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79871073 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43053446 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36827618 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚707355906552319 trainer/global_step‚4649 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4904192504319 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚155776167888896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9998699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43043455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4911350554624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8149596 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚160739583163904 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36827620 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 _runtime‚4038.861813568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚706079085916160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36827620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚706079085916160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚707355548962957 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162427652986880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80700420 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033429504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18334752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709630736969728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43498832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4953797937393 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4961033429504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5013585216512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚713293896368128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10099916 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚714584177317871 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5013494542336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714584534907121 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8235101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718255288176128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18335017 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929797632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚713293896368128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5013494542336 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714584534907121 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8235101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚718255288176128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚713293913407488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18335017 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718254929797632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚713293896368128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚LV/ÆÆ9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162427652986880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80700420 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4961033429504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18334752 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚709630736969728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43498832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4953797937393 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4961033429504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5013585216512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 _step‚384€4Z`±žà‚(:“ ÑíÒÌçåí gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º3Eò½á‚(:² ÑíÒˆ¦–ï disk.rbd0.out‚0 network.sent‚ 7.459135e+06 disk.drbd2165.in‚0$ memory_percent‚96.97554111480713 disk.sda3.out‚89.6875( proc.memory.availableMB‚ 495.52734375$ disk./.usageGB‚589.8976135253906 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 5456.88671875 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda.out‚89.6875 proc.cpu.threads‚17 disk.rbd0.in‚0) disk./.usagePercent‚39.58852380075421 disk.dm-0.in‚ 5456.88671875" disk.dm-0.out‚8600.72119140625 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.out‚10.625 network.recv‚ 1.104131e+06 disk.sda1.in‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 5456.88671875% disk.nvme0n1.out‚8600.72119140625! proc.memory.rssMB‚ 2767.359375* proc.memory.percent‚16.890621185302734 cpu‚25.945899874472577 disk.dm-1.out‚89.6875 disk.rbd1.in‚0' disk.nvme0c0n1.out‚8600.72119140625[fP2â- _wandbruntime‚4080  _runtime‚4080`Ú#ì&ãæM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _runtime‚4080.577227978 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8236846 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37209218 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚385 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18338494 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709777230602240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚713441154437120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713441154437120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718403541839872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚162462102448640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157447587203072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚714731699043465 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954468441225 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5014605912576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚718403183454208 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4962046056448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43507898 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709777247641600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4954808991227 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10101922 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843668020710719e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80717118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157447605815296 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5014515245568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4962387402752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8236843 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4962046056448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18338768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚143880‹ñ&äëM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10101651 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4962046056448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43507898 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚709777247641600 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4954808991227 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10101922 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843668020710719e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80717118 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚157447605815296 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5014515245568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4962387402752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8236843 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4962046056448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18338768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 _runtime‚4080.577227978 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8236846 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37209218 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚385 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18338494 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚709777230602240 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37209220 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚713441154437120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚713441154437120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718403541839872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚162462102448640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚157447587203072 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚714731699043465 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4954468441225 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5014605912576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚713441137397760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚718403183454208 EÁé‚(:¶ àíÒ¼ˆíñ( proc.memory.availableMB‚ 495.58203125* disk./.usagePercent‚39.588530456950096 disk.dm-0.in‚ 5456.88671875" disk.dm-0.out‚8606.69091796875 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.out‚89.96875* proc.memory.percent‚16.890621185302734# disk.nvme0c0n1.in‚ 5456.88671875 disk.dm-1.out‚89.96875 network.sent‚ 7.511402e+06 disk.sda3.in‚0! proc.memory.rssMB‚ 2767.359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚8606.69091796875 cpu‚25.94727541830341 proc.cpu.threads‚17$ disk./.usageGB‚589.8977127075195 disk.rbd1.out‚10.6875 disk.sda2.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚8606.69091796875 disk.rbd0.in‚0 disk.sda.in‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 5456.88671875$ memory_percent‚96.97520732879639 network.recv‚ 1.108271e+06 disk.sda.out‚89.96875 disk.sda1.out‚0 disk.rbd0.out‚0„Áî‘ê‚(:† ïíÒ²¯òî gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï”7úÇë‚(:¼ ïíÒ¦†™ï disk.sda1.out‚0 network.recv‚ 1.112664e+06 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 cpu‚25.948771100071838$ disk./.usageGB‚589.8976097106934 disk.dm-1.out‚ 90.2265625 disk.sda.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.890621185302734 disk.dm-0.in‚ 5456.88671875 network.sent‚ 7.563618e+06 disk.dm-1.in‚0% disk.nvme0n1.out‚8609.85595703125' disk.nvme0c0n1.out‚8609.85595703125 disk.rbd0.in‚0! disk.nvme0n1.in‚ 5456.88671875 disk.rbd1.out‚10.6875# disk.nvme0c0n1.in‚ 5456.88671875 disk.sda.out‚ 90.2265625 disk.sda3.out‚ 90.2265625! proc.memory.rssMB‚ 2767.359375" disk.dm-0.out‚8609.85595703125) disk./.usagePercent‚39.58852354474668 disk.drbd2165.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 495.63671875$ memory_percent‚96.97487354278564 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.out‚0 œ[2ì- _wandbruntime‚4121  _runtime‚4121;²VMÄí¾ _step‚387 % lr-AdamW/pg1‚0.000514135418455331 % lr-AdamW/pg2‚0.000514135418455331 trainer/global_step‚4749 % _timestamp‚1.7843668431198564e+09 _runtime‚ 4121.62577638ïˆíÉîà trainer/global_step‚4749 % _timestamp‚1.7843668431198564e+09 _runtime‚ 4121.62577638 _step‚387 % lr-AdamW/pg1‚0.000514135418455331 % lr-AdamW/pg2‚0.000514135418455331ƒw2ï- _wandbruntime‚4121  _runtime‚4121òãBjrðÜO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8322348 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843668431206505e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943307 ZFDeviceStatsMonitor.on_]í¦;p&train_batch_start/inactive_split_bytes.all.freed‚164150354034176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035159552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159085614983168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5012070270464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37583188 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953284 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18525410 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10203309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720655964889088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43953284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398379 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _runtime‚4121.629829815 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37583188 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5012070270464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720655947849728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37583186 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424301 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10203062 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5064757663232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37583186 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728931328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5064842045952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5011728931328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164150457029120 trainer/global_step‚4749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18525660 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚725668035159552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚716956253523968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676781056JÙ•“ç'ñáO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚725667676781056 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8322348 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843668431206505e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43943307 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164150354034176 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚725668035159552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159085614983168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5012070270464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37583188 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43953284 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18525410 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10203309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159085596370944 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720655964889088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43953284 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 102994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚721960327398379 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _runtime‚4121.629829815 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37583188 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5012070270464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚388 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720655947849728 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37583186 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5004414424301 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10203062 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5064757663232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚716956270563328 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37583186 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5011728931328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5064842045952 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5011728931328 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164150457029120 trainer/global_step‚4749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18525660 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚725668035159552 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚716956253523968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250„ðCVÁ2ò- _wandbruntime‚4122  _runtime‚4122ÁÚPì&óæM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37590818 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37590818 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543177 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10205420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5065781123072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159119025129472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37590820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164184806252544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159119043741696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚725816288823296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10205148 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717102781235200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928133 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741558272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚4749 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5013082904576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8324093 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843668437492278e+09 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478135 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18529241 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553170 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18529516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164184913434624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚720803188879360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _step‚389 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8324096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5005425478135 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18529241 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81553170 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18529516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164184913434624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚720803188879360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 _step‚389 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8324096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37590818 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81543177 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10205420 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5065781123072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720803188879360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159119025129472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37590820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164184806252544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159119043741696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚725816288823296 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43962350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81543177 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10205148 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚717102781235200 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5005084928133 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5012741558272 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 trainer/global_step‚4749 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5013082904576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8324093 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843668437492278e+09 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728018016370688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5055030911209 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 4163.13672387 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165873158035968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5062424433152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚729336835068137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚733080782142976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729336477478887 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44407736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚733080782142976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚728017999331328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚724281787117568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37964788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10307531 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18716871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5116014452224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44397759 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5062765772288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44397759 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8409598 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423764480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5116105126400Õ×B@Ö"ƒàO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5055030911209 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 4163.13672387 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165873158035968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5062424433152 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚729336835068137 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚733080782142976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚729336477478887 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44407736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚733080782142976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚728017999331328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚724281787117568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37964788 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10307531 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚4799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18716871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5116014452224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44397759 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5062765772288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44397759 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8409598 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚733080423764480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bU£ytes.small_pool.allocated‚ 5116105126400 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37964786 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44407736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚733080423764480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚728018016370688 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5062765772288 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728018016370688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508ˆâKž 2„- _wandbruntime‚4163  _runtime‚4163¿Ô9ì&…æM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5063778406400 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5056041965043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379229 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406811 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚160790481668096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚728165257400320 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚724428314828800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚4163.764614513 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5063437060096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843668852584417e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406811 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚733228677421056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055701415041 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚729483999204481 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44416802 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437060096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18720663 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37972420 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5117123764224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37972418 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165907496153088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117033097216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚729484356793843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728165257400320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724428297789440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚733229035806720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10309591 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843668852584417e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44406811 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚733228677421056 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5055701415041 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82389222 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚729483999204481 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44416802 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5063437060096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18720663 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37972420 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5117123764224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37972418 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8411343 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165907496153088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5117033097216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚729484356793843 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728165257400320 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚724428297789440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚733229035806720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10309591 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379229 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44406811‰–<A2‡- _wandbruntime‚4163  _runtime‚4163|ÂWˆ‚ 2loss/SRqqt1/assignment_loss‚0.6077895011497935 /loss/FBt2/detection_loss‚1.1859572577807638 0loss/FRt1/assignment_loss‚1.9387332119638958 1loss/FBt2/assignment_loss‚0.02435218121161916 %loss/total_loss‚9.163891710472274 epoch‚11 1loss/FBt1/assignment_loss‚0.03670621653302546 /loss/FRt2/detection_loss‚1.3818530344333306 1loss/SRqqt1/detection_loss‚0.8472679097060885 1loss/SRqqt2/detection_loss‚0.6795307322829266 /loss/FBt1/detection_loss‚0.4800255957467497 trainer/global_step‚4799 $ _timestamp‚1.784366885260052e+09 _runtime‚4163.765741151 0loss/FRt2/assignment_loss‚0.9047215148861967 3loss/SRqqt2/assignment_loss‚0.45960808539483433 /loss/FRt1/detection_loss‚0.6173464693830519 _step‚394si ‰‡ 2loss/SRqqt1/assignment_loss‚0.6077895011497935 /loss/FBt2/detection_loss‚1.1859572577807638 0loss/FRt1/assignment_loss‚1.9387332119638958 1loss/FBt2/assignment_loss‚0.02435218121161916 %loss/total_loss‚9.163891710472274 epoch‚11 1loss/FBt1/assignment_loss‚0.03670621653302546 /loss/FRt2/detection_loss‚1.3818530344333306 1loss/SRqqt1/detection_loss‚0.8472679097060885 1loss/SRqqt2/detection_loss‚0.6795307322829266 /loss/FBt1/detection_loss‚0.4800255957467497 trainer/global_step‚4799 $ _timestamp‚1.784366885260052e+09 _runtime‚4163.765741151 0loss/FRt2/assignment_loss‚0.9047215148861967 3loss/SRqqt2/assignment_loss‚0.45960808539483433 /loss/FRt1/detection_loss‚0.6173464693830519 _step‚394Š-5‹“‘Š‚(:† «ŽíÒìÿªï gpu.0.gpu‚1.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.231' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ¸Ä%Ä‹‚(:¹ «ŽíÒ”ùöð network.sent‚ 7.640513e+06 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 5546.51953125! disk.dm-0.out‚8624.6650390625 disk.sda1.in‚0" proc.memory.rssMB‚ 2769.9921875* disk./.usagePercent‚39.588539417213795 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0$ memory_percent‚96.99106216430664) proc.memory.percent‚16.90669059753418 network.recv‚ 1.126842e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 5546.51953125& disk.nvme0c0n1.out‚8624.6650390625 disk.drbd2165.in‚0$ disk.nvme0n1.out‚8624.6650390625 disk.sda.out‚ 91.38671875 proc.cpu.threads‚17 disk.rbd1.out‚10.875 disk.sda.in‚0 disk.sda1.out‚0$ disk./.usageGB‚589.8978462219238 disk.dm-1.out‚ 91.38671875! disk.nvme0n1.in‚ 5546.51953125 disk.sda2.out‚0 disk.sda3.out‚ 91.38671875& proc.memory.availableMB‚ 492.984375 cpu‚25.95937474718361bõÃ2Œ- _wandbruntime‚4181  _runtime‚4181«°lÖÆ ž{ & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020797017961740494 *jet/accuracy_2_of_2‚0.1521417796611786 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6216589212417603 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.027956798672676086 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.295390248298645 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8991091847419739 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1407335102558136 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3902302384376526 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23467686772346497 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.8021720051765442 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41865280270576477 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2495768815279007 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6880595088005066 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8765683770179749 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6471558809280396 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943804144859314 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953286051750183 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4068589508533478 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891344904899597 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 trainer/global_step‚4799 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_1‚0.3357619047164917 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3869118094444275 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666197776794 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.719588577747345 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8050372004508972 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5982068777084351 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08978457748889923 (particle/accuracy‚0.8451683521270752 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08990971744060516 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6480554342269897 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2227720320224762 *jet/accuracy_1_of_3‚0.9253703355789185 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9749104380607605 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8646258115768433 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6909388303756714 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 6validation_average_jet_accuracy‚0.3780277669429779 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7430540919303894 _step‚395 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.673554539680481 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6504985094070435 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.052969496697187424 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9397505521774292 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07579716295003891 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚8.963556090950172 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.652721643447876 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20703250169754028 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 _runtime‚4181.972194078 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47072625160217285 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09753721952438354 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7150267958641052 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666567325592 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2471419721841812 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_2‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20693881809711456 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22625572979450226 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6397719383239746 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.693617045879364 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18166068196296692 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19007334113121033 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23467686772346497 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2514623999595642 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07320531457662582 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15341749787330627 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7407407164573669 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9874552488327026 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4910447299480438 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6869693398475647 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6280818581581116 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7325968146324158 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5615093111991882 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3541666567325592 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & ¯{TàÞPurity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2906675338745117 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9048901796340942 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5666711330413818 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2466711699962616 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961695075035095 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8737195134162903 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1567552238702774 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2316974401473999 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.6267738342285156 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9832243919372559 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18503901362419128 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6053968667984009 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808861017227173 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.880442202091217 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_2_of_2‚0.9998962879180908 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.650712788105011 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 jet/accuracy_3_of_5‚1 epoch‚11 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8715797662734985 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.652721643447876 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7464342713356018 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.878265917301178 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09065764397382736 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2063121795654297 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_4_of_4‚0.939541220664978 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4934016168117523 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6213576197624207 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6344389915466309 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8801019787788391 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.774177074432373 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7531477808952332 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3273985683917999 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6742194294929504 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9174059629440308 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.734207272529602 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.040513958781957626 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31281423568725586 % _timestamp‚1.7843669034655821e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7914965748786926 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0390695258975029 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6885665059089661 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48668062686920166 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09896053373813629 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6412626504898071 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6903494596481323 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21931973099708557 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6254121661186218 *jet/accuracy_2_of_3‚0.5539262890815735 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23458510637283325 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7463204264640808 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7716937065124512 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8050372004508972 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991891980171204 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24646686017513275 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2196798324584961 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988061189651489 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.328125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16990594565868378 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7199119925498962 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05201470106840134 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23263880610466003 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6435191631317139 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07320531457662582 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939055442810059 particle/accuracy_2_of_3‚1 /particle/accuracy_3_of_3‚0.9994933605194092 'particle/f_score‚0.6348358988761902 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951714873313904 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1792747676372528 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7381551265716553 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7391039729118347 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930726885795593 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *jet/accuracy_1_of_4‚0.9670750498771667 *jet/accuracy_2_of_4‚0.7783169746398926 particle/accuracy_5_of_5‚0ÿ/©=Ž£{ =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2495768815279007 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6880595088005066 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8765683770179749 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6471558809280396 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943804144859314 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9953286051750183 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4068589508533478 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891344904899597 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 trainer/global_step‚4799 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 *jet/accuracy_1_of_1‚0.3357619047164917 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.14814814925193787 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3869118094444275 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666197776794 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.719588577747345 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8050372004508972 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5982068777084351 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08978457748889923 (particle/accuracy‚0.8451683521270752 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08990971744060516 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6480554342269897 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2227720320224762 *jet/accuracy_1_of_3‚0.9253703355789185 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9749104380607605 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8646258115768433 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6909388303756714 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11916667222976685 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 6validation_average_jet_accuracy‚0.3780277669429779 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7430540919303894 _step‚395 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.673554539680481 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6504985094070435 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.052969496697187424 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9397505521774292 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4930555522441864 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.07579716295003891 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚8.963556090950172 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.652721643447876 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20703250169754028 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 _runtime‚4181.972194078 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7407407164573669 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47072625160217285 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09753721952438354 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7150267958641052 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666567325592 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2471419721841812 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 particle/accuracy_1_of_2‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20693881809711456 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22625572979450226 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6397719383239746 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_4_of_5‚0 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.693617045879364 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18166068196296692 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19007334113121033 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23467686772346497 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2514623999595642 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07320531457662582 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15341749787330627 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7407407164573669 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9874552488327026 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4910447299480438 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6869693398475647 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6280818581581116 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7325968146324158 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5615093111991882 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3541666567325592 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2906675338745117 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.9048901796340942 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5666711330413818 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2466711699962616 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961695075035095 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8737195134162903 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1567552238702774 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259700775146 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2316974401473999 particle/accuracy_3_of_4‚1 +particle/sensitivity‚0.6267738342285156 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9832243919372559 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18503901362419128 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6053968667984009 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808861017227173 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.880442202091217 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 /particle/accuracy_2_of_2‚0.9998962879180908 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.650712788105011 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 jet/accuracy_3_of_5‚1 epoch‚11 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8715797662734985 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.652721643447876 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7464342713356018 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.878265917301178 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09065764397382736 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2063121795654297 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_4_of_4‚0.939541220664978 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4934016168117523 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6213576197624207 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6344389915466309 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8801019787788391 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.774177074432373 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7531477808952332 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_1_of_4‚1 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3273985683917999 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6742194294929504 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9174059629440308 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.734207272529602 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.040513958781957626 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.31281423568725586 % _timestamp‚1.7843669034655821e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7914965748786926 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0390695258975029 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6885665059089661 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48668062686920166 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09896053373813629 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6412626504898071 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6903494596481323 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21931973099708557 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6254121661186218 *jet/accuracy_2_of_3‚0.5539262890815735 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23458510637283325 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7463204264640808 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7716937065124512 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8050372004508972 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9991891980171204 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24646686017513275 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2196798324584961 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988061189651489 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.328125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16990594565868378 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9912398457527161 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7199119925498962 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05201470106840134 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23263880610466003 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6435191631317139 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07320531457662582 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939055442810059 particle/accuracy_2_of_3‚1 /particle/accuracy_3_of_3‚0.9994933605194092 'particle/f_score‚0.6348358988761902 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951714873313904 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1792747676372528 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7381551265716553 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7391039729118347 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958134651184 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930726885795593 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *jet/accuracy_1_of_4‚0.9670750498771667 *jet/accuracy_2_of_4‚0.7783169746398926 particle/accuracy_5_of_5‚0 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.020797017961740494 *jet/accuracy_2_of_2‚0.1521417796611786 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6216589212417603 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.14814814925193787 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.027956798672676086 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.295390248298645 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8991091847419739 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1407335102558136 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3902302384376526 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23467686772346497 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.8021720051765442 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41865280270576477‹H`“‚(:ˆ ºŽíÒ ÁŠî gpu.0.gpu‚77.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚160.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mzª-¿‚(:´ ºŽíÒ™¾ ð disk.rbd1.out‚10.9375 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2784.53515625 network.recv‚ 1.131245e+06 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 91.68359375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# memory_percent‚95.4719066619873% disk.nvme0c0n1.out‚8875.197265625 disk.sda3.out‚ 91.68359375 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 5627.8046875 disk.sda.out‚ 91.68359375 disk.sda1.out‚0 cpu‚25.96944546787571 proc.cpu.threads‚17' proc.memory.availableMB‚ 741.8828125$ disk./.usageGB‚589.8989486694336 disk.nvme0n1.in‚ 5627.8046875 disk.dm-0.out‚8875.197265625 disk.sda2.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚16.99545383453369) disk./.usagePercent‚39.58861340339119# disk.nvme0n1.out‚8875.197265625 disk.dm-0.in‚ 5627.8046875 network.sent‚ 7.703418e+06‰w§‘‘‚(:† ÉŽíÒùÚí gpu.0.gpu‚29.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú=¾¬’‚(:¡ ÉŽíÒ­¹Žö network.sent‚ 7.760697e+06 disk.sda.in‚0 network.recv‚ 1.135658e+06 disk.sda1.out‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2784.53515625% disk.nvme0n1.out‚8878.82763671875 disk.sda2.out‚0' disk.nvme0c0n1.out‚8878.82763671875( proc.memory.availableMB‚ 742.05078125 proc.cpu.threads‚17$ disk./.usageGB‚589.8990097045898 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚10.9375" disk.nvme0c0n1.in‚ 5627.8046875 disk.sda.out‚92 disk.dm-1.in‚0 disk.drbd2165.out‚0" disk.dm-0.out‚8878.82763671875 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚95.47088146209717) disk./.usagePercent‚39.58861749951174 disk.rbd0.in‚0 disk.dm-0.in‚ 5627.8046875 disk.sda3.out‚92 disk.dm-1.out‚92 disk.nvme0n1.in‚ 5627.8046875) proc.memory.percent‚16.99545383453369 cpu‚25.97107638020816 disk.drbd2165.in‚0ÕSzø‘“‚(:† ØŽíÒâÅÿî gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÖ”‚(:· ØŽíÒ’À¼ð disk.sda.in‚0$ memory_percent‚95.47162055969238 cpu‚25.97231238829193 proc.cpu.threads‚17 network.recv‚ 1.138738e+06 disk.rbd1.in‚0# proc.memory.rssMB‚ 2784.53515625* disk./.usagePercent‚39.588540697251474 disk.sda3.in‚0 disk.dm-1.out‚ 92.19921875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 5627.8046875" disk.dm-0.out‚9129.35498046875 disk.rbd1.out‚11" disk.nvme0c0n1.in‚ 5627.8046875) proc.memory.percent‚16.99545383453369 disk.rbd0.out‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚9129.35498046875 disk.sda.out‚ 92.19921875 disk.dm-1.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 741.9296875 disk.drbd2165.out‚0 disk.dm-0.in‚ 5627.8046875 disk.sda1.in‚0$ disk./.usageGB‚589.8978652954102 disk.sda3.out‚ 92.19921875 disk.drbd2165.in‚0 disk.sda2.in‚0 network.sent‚ 7.765623e+06% disk.nvme0n1.out‚9129.35498046875Yò.2•- _wandbruntime‚4224  _runtime‚4224¹ÓãÇ–Á trainer/global_step‚4849 % _timestamp‚1.7843669464339905e+09 _runtime‚4224.940027825 _step‚396 & lr-AdamW/pg1‚0.0005111123682649759 & lr-AdamW/pg2‚0.0005111123682649759ë‚ÀoÌ—Æ _runtime‚4224.940027825 _step‚396 & lr-AdamW/pg1‚0.0005111123682649759 & lr-AdamW/pg2‚0.0005111123682649759 trainer/global_step‚4849 % _timestamp‚1.7843669464339905e+09Œ˜­¾P2˜- _wandbruntime‚4224  _runtime‚4224Æ]I«E™ÛO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5130853143040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742440470606336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5185439313408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚737309617463296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38467052 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38467054 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44932515 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389590 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5131194482176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8499368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162444402935808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18942679 trainer/global_step‚4849 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5123367123351 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large ™àœ_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚737309617463296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚737309634502656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5131194482176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38467054 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843669464351926e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10443561 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733515983925248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167629922437632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738639368087959 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470606336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5123026573461 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8499371 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4224.941303765 _step‚397 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18942932 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚737309634502656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5130853143040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162444384323584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742440828984832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0<ôCÖæ'šàO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742440828984832 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8499368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚162444402935808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18942679 trainer/global_step‚4849 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5123367123351 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚737309617463296 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚737309634502656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5131194482176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38467054 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 % _timestamp‚1.7843669464351926e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10443561 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚733515983925248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167629922437632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚738639368087959 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742440470606336 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5123026573461 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8499371 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4224.941303765 _step‚397 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18942932 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚737309634502656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5130853143040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚162444384323584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742440828984832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5130853143040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742440470606336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5185439313408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚737309617463296 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38467052 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38467054 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44932515 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389590 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5131194482176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5131865769984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5186549364224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162477831694336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843669470733008e+09 _step‚398 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738786532224303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18946772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406274 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44931590 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10445389 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737456858492928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44941581 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167664275973632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 _runtime‚4225.579467079 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚737456858492928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10445656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44931590 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44941581 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162477813082112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5124037627183 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5131865769984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737456875532288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18946502 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚4849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚738786889813665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8501116 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38474686 _runtime‚4225.579467079 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38474686 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8501113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚737456858492928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10445656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44931590 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44941581 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162477813082112 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5124037627183 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5131865769984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚737456875532288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18946502 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚4849 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚738786889813665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8501116 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5131865769984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5186549364224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162477831694336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843669470733008e+09 _step‚398 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738786532224303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeÙÝu¤´ viceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18946772 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83406274 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44931590 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10445389 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚737456858492928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44941581 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚167664275973632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438Ž Hì×2ž- _wandbruntime‚4225  _runtime‚4225("{õŸ 3loss/SRqqt2/assignment_loss‚0.48179640347505187 /loss/FRt2/detection_loss‚1.3839765010851333 1loss/SRqqt2/detection_loss‚0.6150144962748515 trainer/global_step‚4849 % _timestamp‚1.7843669470745287e+09 _runtime‚4225.580366017 /loss/FRt1/assignment_loss‚1.912427839915918 1loss/FBt1/assignment_loss‚0.00622360969556561 1loss/SRqqt1/detection_loss‚0.8140757183425675 _step‚399 0loss/FRt2/assignment_loss‚0.8780892355360459 1loss/FBt2/assignment_loss‚0.03429701651168532 /loss/FRt1/detection_loss‚0.5297795618694388 /loss/FBt1/detection_loss‚0.3609209253818595 /loss/FBt2/detection_loss‚0.6178465423333209 2loss/SRqqt1/assignment_loss‚0.6102824098021322 $loss/total_loss‚8.24473026022357 epoch‚12=&Y¼  † 3loss/SRqqt2/assignment_loss‚0.48179640347505187 /loss/FRt2/detection_loss‚1.3839765010851333 1loss/SRqqt2/detection_loss‚0.6150144962748515 trainer/global_step‚4849 % _timestamp‚1.7843669470745287e+09 _runtime‚4225.580366017 /loss/FRt1/assignment_loss‚1.912427839915918 1loss/FBt1/assignment_loss‚0.00622360969556561 1loss/SRqqt1/detection_loss‚0.8140757183425675 _step‚399 0loss/FRt2/assignment_loss‚0.8780892355360459 1loss/FBt2/assignment_loss‚0.03429701651168532 /loss/FRt1/detection_loss‚0.5297795618694388 /loss/FBt1/detection_loss‚0.3609209253818595 /loss/FBt2/detection_loss‚0.6178465423333209 2loss/SRqqt1/assignment_loss‚0.6102824098021322 $loss/total_loss‚8.24473026022357 epoch‚12lñÖ“¡‚(:ˆ çŽíҽϊí gpu.0.gpu‚57.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.457' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓôÒÀ¢‚(:µ çŽíÒ䚧ð disk.dm-1.in‚0 disk.dm-1.out‚ 92.41015625 disk.drbd2165.in‚0" disk.dm-0.out‚9133.14013671875) proc.memory.percent‚16.99545383453369 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 network.sent‚ 7.770573e+06) disk./.usagePercent‚39.58853634512339 disk.dm-0.in‚ 5627.8046875 proc.cpu.threads‚17 disk.rbd1.out‚11 disk.sda.in‚0 disk.sda.out‚ 92.41015625 disk.rbd0.out‚0 network.recv‚ 1.142168e+06$ disk./.usageGB‚589.8978004455566" disk.nvme0c0n1.in‚ 5627.8046875% disk.nvme0n1.out‚9133.14013671875& proc.memory.availableMB‚ 741.921875 disk.sda1.in‚0 disk.sda3.out‚ 92.41015625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2784.53515625 cpu‚25.973587377411032 disk.sda2.in‚0' disk.nvme0c0n1.out‚9133.14013671875 disk.nvme0n1.in‚ 5627.8046875# memory_percent‚95.4716682434082 disk.rbd1.in‚0 disk.sda3.in‚0…î¬\ž£‚(:“ öŽíÒˆŒ›í gpu.0.gpu‚43.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó= \Ȥ‚(:½ öŽíÒ¡ÝÀï disk.sda1.in‚0 disk.sda3.out‚ 92.66796875 disk.dm-1.in‚0! disk.dm-0.out‚9138.8935546875( proc.memory.availableMB‚ 741.61328125$ disk./.usageGB‚589.8978118896484 disk.sda.out‚ 92.66796875 disk.sda1.out‚0 disk.drbd2165.out‚0# memory_percent‚95.4735517501831 cpu‚25.974919709851328 proc.cpu.threads‚17) disk./.usagePercent‚39.58853711314599 disk.rbd1.out‚11.0625 disk.rbd0.in‚0& disk.nvme0c0n1.out‚9138.8935546875 disk.dm-1.out‚ 92.66796875 disk.rbd1.in‚0 disk.sda.in‚0$ disk.nvme0n1.out‚9138.8935546875 network.recv‚ 1.146436e+06 disk.sda2.in‚0 disk.dm-0.in‚ 5716.24609375 network.sent‚ 7.833387e+06 disk.sda2.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 5716.24609375* proc.memory.percent‚16.995835304260254# disk.nvme0c0n1.in‚ 5716.24609375 disk.sda3.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2784.59765625ÒûK\¥‚(:… …íÒŒ„áí gpu.0.gpu‚26.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù°¬^˦‚(:À …íÒúƒãï disk.dm-1.out‚ 92.97265625 disk.rbd0.out‚0( proc.memory.availableMB‚ 495.58984375 network.recv‚ 1.150145e+06! disk.nvme0n1.in‚ 5716.24609375% disk.nvme0n1.out‚9142.87744140625 disk.dm-0.in‚ 5716.24609375" disk.dm-0.out‚9142.87744140625 disk.sda2.out‚0 disk.rbd1.out‚11.0625) disk./.usagePercent‚39.58861340339119# disk.nvme0c0n1.in‚ 5716.24609375 disk.dm-1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2784.59765625 proc.cpu.threads‚17 disk.sda.out‚ 92.97265625 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 network.sent‚ 7.838422e+06 disk.sda.in‚0 disk.sda3.out‚ 92.97265625 cpu‚25.97628770614279$ memory_percent‚96.97515964508057 disk.rbd1.in‚0' disk.nvme0c0n1.out‚9142.87744140625* proc.memory.percent‚16.995835304260254$ disk./.usageGB‚589.8989486694336 disk.sda2.in‚0¤µ×Ë2§- _wandbruntime‚4266  _runtime‚4266Ðú¬ ƨÀ $ _timestamp‚1.784366987965814e+09 _runtime‚4266.471795129 _step‚400 & lr-AdamW/pg1‚0.0005095795893849309 & lr-AdamW/pg2‚0.0005095795893849309 trainer/global_step‚4899ãË©Å & lr-AdamW/pg1‚0.0005095795893849309 & lr-AdamW/pg2‚0.0005095795893849309 trainer/global_step‚4899 $ _timestamp‚1.784366987965814e+09 _runtime‚4266.471795129 _step‚400&A›µ2ª- _wandbruntime‚4266  _runtime‚42666¶WDâ'«ÜO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181548644864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5236788071424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _runtime‚4266.473064476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225642 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181548644864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38848652 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169352521744384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38848654 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740841534558208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169352628933632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8586618 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217589760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5173643060369 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚749853575968256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10547239 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45376990 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45386967 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19133857 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5236699494400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚746015518168467 _step‚401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181889984000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38848654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45386967 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚740841517518848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744671685984256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872”ÀTfç'¬áO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 % _timestamp‚1.7843669879669065e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19134112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164115822249984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5181548644864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5236788071424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 _runtime‚4266.473064476 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225642 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5181548644864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38848652 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169352521744384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38848654 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚740841534558208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169352628933632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8586618 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚749853217589760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5173643060369 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚749853575968256 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744671668944896 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10547239 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38848652 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45376990 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45386967 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744671685984256 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19133857 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5236699494400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚746015518168467 _step‚401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5181889984000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38848654 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45386967 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚740841517518848 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744671685984256 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10549328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚746163AÉ*f039894173 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚750001829632000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744818909974528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45386042 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5182561271808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5237809582592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38856284 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396033 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84242326 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114091 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5237714721280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚740988028190720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829632000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8588363 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45386042 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19137691 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242326 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚746162682304811 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252319 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10549592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8588366 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396033 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0*lp|ð&¯êM RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10549328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚746163039894173 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744818909974528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚750001829632000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744818909974528 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45386042 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5182561271808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38856284 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5237809582592 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38856284 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45396033 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84242326 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5174654114091 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5237714721280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚740988028190720 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750001829632000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚740988045230080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8588363 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744818927013888 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45386042 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19137691 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242326 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚746162682304811 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252319 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10549592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8588366 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45396033 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 _runtime‚4267.105001053 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38856286’ò£b[2°- _wandbruntime‚4267  _runtime‚4267{“d±‚ trainer/global_step‚4899 3loss/SRqqt2/assignment_loss‚0.46859514104249916 2loss/FBt1/assignment_loss‚0.003909903163642979 .loss/FBt1/detection_loss‚0.489520736469128 _step‚403 1loss/SRqqt2/detection_loss‚0.6840169654897756 % _timestamp‚1.7843669886004598e+09 0loss/FRt1/assignment_loss‚1.9949401210412723 0loss/FRt2/assignment_loss‚0.9176478770234502 /loss/FRt1/detection_loss‚0.6066055447085933 /loss/FRt2/detection_loss‚1.4026765387312383 .loss/FBt2/detection_loss‚0.679706866788264 epoch‚12 _runtime‚4267.106141196 1loss/SRqqt1/assignment_loss‚0.576207141504208 2loss/FBt2/assignment_loss‚0.003059342846788956 1loss/SRqqt1/detection_loss‚1.0283875720902778 %loss/total_loss‚8.855273750899139¿ùɳ ²‡ % _timestamp‚1.7843669886004598e+09 0loss/FRt1/assignment_loss‚1.9949401210412723 0loss/FRt2/assignment_loss‚0.9176478770234502 /loss/FRt1/detection_loss‚0.6066055447085933 /loss/FRt2/detection_loss‚1.4026765387312383 .loss/FBt2/detection_loss‚0.679706866788264 epoch‚12 _runtime‚4267.106141196 1loss/SRqqt1/assignment_loss‚0.576207141504208 2loss/FBt2/assignment_loss‚0.003059342846788956 1loss/SRqqt1/detection_loss‚1.0283875720902778 %loss/total_loss‚8.855273750899139 trainer/global_step‚4899 3loss/SRqqt2/assignment_loss‚0.46859514104249916 2loss/FBt1/assignment_loss‚0.003909903163642979 .loss/FBt1/detection_loss‚0.489520736469128 _step‚403 1loss/SRqqt2/detection_loss‚0.6840169654897756“U,a‘³‚(:† ”íÒÑõ±î gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.401' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6802 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãó3ÊÈ´‚(:½ ”íÒõß‹ï disk.sda1.out‚0% disk.nvme0n1.out‚9149.48486328125 disk.sda.out‚ 93.44140625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚16.995835304260254 cpu‚25.97754193979656 proc.cpu.threads‚17 disk.rbd1.out‚11.125" disk.dm-0.out‚9149.48486328125# proc.memory.rssMB‚ 2784.59765625 network.sent‚ 7.843261e+06 disk.sda3.out‚ 93.44140625 disk.dm-1.out‚ 93.44140625 disk.rbd0.in‚0 disk.sda2.in‚0$ memory_percent‚96.97332382202148& proc.memory.availableMB‚ 495.890625$ disk./.usageGB‚589.8978729248047 disk.rbd1.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 5716.24609375 disk.sda2.out‚0) disk./.usagePercent‚39.58854120926654# disk.nvme0c0n1.in‚ 5716.24609375 disk.sda3.in‚0 network.recv‚ 1.153391e+06' disk.nvme0c0n1.out‚9149.48486328125 disk.drbd2165.in‚0 disk.dm-0.in‚ 5716.24609375 disk.sda.in‚0h¶D“µ‚(:ˆ £íÒʯÄí gpu.0.gpu‚68.0 gpu.0.memory‚18+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ¾É;¶¶‚(:« £íÒÙ¤³ï disk.sda.in‚0 disk.sda1.in‚0) disk./.usagePercent‚39.58855016953024 disk.nvme0n1.out‚ 9153.09375 disk.dm-0.in‚ 5716.24609375 disk.sda2.out‚0 network.recv‚ 1.158159e+06 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 495.890625* proc.memory.percent‚16.995835304260254 network.sent‚ 7.906431e+06 disk.sda3.out‚ 93.78515625# proc.memory.rssMB‚ 2784.59765625 cpu‚25.978948714232637 disk.dm-1.in‚0$ memory_percent‚96.97332382202148 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 5716.24609375! disk.nvme0n1.in‚ 5716.24609375 disk.dm-1.out‚ 93.78515625 proc.cpu.threads‚17# disk./.usageGB‚589.898006439209 disk.rbd0.out‚0 disk.rbd1.out‚11.125 disk.sda.out‚ 93.78515625 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.dm-0.out‚ 9153.09375 disk.rbd1.in‚0! disk.nvme0c0n1.out‚ 9153.09375ÿåež·‚(:“ ²íÒñŠ«í gpu.0.gpu‚39.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ýþ6½¸‚(:² ²íÒ”Ñïð disk.sda.out‚94.09375' proc.memory.availableMB‚ 497.2265625 disk.dm-1.out‚94.09375 disk.rbd0.in‚0! disk.nvme0n1.in‚ 5770.08984375& disk.nvme0c0n1.out‚9158.4736328125 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚11.25 network.recv‚ 1.161608e+06$ disk./.usageGB‚589.8977584838867 disk.sda3.in‚0 cpu‚25.980158154504775 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 5770.08984375* proc.memory.percent‚16.995835304260254 disk.sda1.in‚0 disk.sda3.out‚94.09375 disk.drbd2165.in‚0$ disk.nvme0n1.out‚9158.4736328125 network.sent‚ 7.911299e+06) disk./.usagePercent‚39.58853352904051 disk.sda1.out‚0 disk.dm-0.in‚ 5770.08984375 disk.rbd1.in‚0$ memory_percent‚96.96516990661621! disk.dm-0.out‚9158.4736328125 disk.sda2.out‚0# proc.memory.rssMB‚ 2784.59765625ø p/2¹- _wandbruntime‚4308  _runtime‚43089œÞ®Äº¾ _step‚404 % lr-AdamW/pg1‚0.000508032772275681 % lr-AdamW/pg2‚0.000508032772275681 trainer/global_step‚4949 $ _timestamp‚1.784367029688905e+09 _runtime‚4308.1948248917çɻà _step‚404 % lr-AdamW/pg1‚0.000508032772275681 % lr-AdamW/pg2‚0.000508032772275681 trainer/global_step‚4949 $ _timestamp‚1.784367029688905e+09 _runtime‚4308.194824891”d¹Ûð2¼- _wandbruntime‚4308  _runtime‚43080z, &½ÜO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165787278788608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚757266322951680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45841419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248975 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚4949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585485824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45841419 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39230252 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8673871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚753391310659725 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39230254 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752033720426496 _step‚405 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚752033737465856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39230254 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5232244146688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964573184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757265964573184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171075101994496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5224259547277 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752033720426496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224600097167 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061694 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831442 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45831442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10651516 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5287930395136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19325134 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165787260176384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757266322951680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071673 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BaÂ-Ýnitor.on_train_batch_start/segment.small_pool.allocated‚269 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752033737465856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5232585485824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438õ{mç'¾áO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚757265964573184 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚757265964573184 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚171075101994496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5224259547277 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752033720426496 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224600097167 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061694 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚748167068151808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45831442 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45831442 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10651516 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5287930395136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19325134 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165787260176384 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚757266322951680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071673 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165787278788608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚757266322951680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45841419 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753391668248975 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚4949 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5232585485824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45841419 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39230252 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8673871 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚753391310659725 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39230254 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752033720426496 _step‚405 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚752033737465856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39230254 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5232244146688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0•ÎBº2¿- _wandbruntime‚4308  _runtime‚4308þ!³iì&ÀæM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39237886 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165820707547136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚406 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚757414576615424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚748313578823680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚4949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5233598119936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5233256773632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19328976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8675613 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚757414218229760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974681 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225611151001 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚757414576615424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752180978495488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45850485 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256773632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10653363 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171109662242816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983+q¥öñ&ÁëM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39237884 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚757414576615424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚752180978495488 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45850485 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5233256773632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088371 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752180978495488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10653363 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171109662242816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39237886 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165820707547136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚282 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚406 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚757414576615424 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚279 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚748313578823680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 trainer/global_step‚4949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5233598119936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5233256773632 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19328976 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8675613 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚757414218229760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753539189974681 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225611151001 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976–c#Ó¹2Â- _wandbruntime‚4308  _runtime‚4308ádò Ä 3loss/FBt1/assignment_loss‚0.0016418851041567514 2loss/FBt2/assignment_loss‚0.004304780335080837 /loss/FRt1/detection_loss‚0.5668491462871954 1loss/SRqqt1/detection_loss‚1.0641462610583403 trainer/global_step‚4949 % _timestamp‚1.7843670303216655e+09 1loss/SRqqt2/assignment_loss‚0.533176466820183 /loss/FBt1/detection_loss‚0.4112101330616939 /loss/FBt2/detection_loss‚1.0897239312333495 %loss/total_loss‚9.233835272482597 epoch‚12 _runtime‚4308.827340238 0loss/FRt2/assignment_loss‚0.9033666350508249 2loss/SRqqt1/assignment_loss‚0.6707550408647762 1loss/SRqqt2/detection_loss‚0.6802792829224116 0loss/FRt1/assignment_loss‚1.9126838969251823 /loss/FRt2/detection_loss‚1.3956978128194029 _step‚407…KˆÄ‰ 0loss/FRt1/assignment_loss‚1.9126838969251823 /loss/FRt2/detection_loss‚1.3956978128194029 _step‚407 3loss/FBt1/assignment_loss‚0.0016418851041567514 2loss/FBt2/assignment_loss‚0.004304780335080837 /loss/FRt1/detection_loss‚0.5668491462871954 1loss/SRqqt1/detection_loss‚1.0641462610583403 trainer/global_step‚4949 % _timestamp‚1.7843670303216655e+09 1loss/SRqqt2/assignment_loss‚0.533176466820183 /loss/FBt1/detection_loss‚0.4112101330616939 /loss/FBt2/detection_loss‚1.0897239312333495 %loss/total_loss‚9.233835272482597 epoch‚12 _runtime‚4308.827340238 0loss/FRt2/assignment_loss‚0.9033666350508249 2loss/SRqqt1/assignment_loss‚0.6707550408647762 1loss/SRqqt2/detection_loss‚0.6802792829224116—)âÖ“Å‚(:ˆ ÁíÒé£þí gpu.0.gpu‚97.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Grw= Æ‚(:Åôož¶´ ÁíÒ•óˆñ disk.dm-1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 496.93359375 disk.rbd1.in‚0 disk.rbd1.out‚11.25 disk.sda3.in‚0 disk.dm-1.out‚ 94.6796875 disk.dm-0.in‚ 5782.69140625* proc.memory.percent‚16.995835304260254 network.sent‚ 7.930003e+06 disk.sda.out‚ 94.6796875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0n1.out‚9161.595703125 disk.sda3.out‚ 94.6796875 disk.dm-0.out‚9161.595703125 network.recv‚ 1.1653e+06$ disk./.usageGB‚589.8980484008789 disk.sda1.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 5782.69140625) disk./.usagePercent‚39.58855298561311 disk.sda.in‚0# disk.nvme0c0n1.in‚ 5782.69140625$ memory_percent‚96.96695804595947 proc.cpu.threads‚17% disk.nvme0c0n1.out‚9161.595703125 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2784.59765625 cpu‚25.981259774258827ë'‰œÇ‚(:‘ ÐíÒûë›í gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AMÑÊÈ‚(:¿ ÐíÒ¤Èëð' disk.nvme0c0n1.out‚9167.62353515625$ memory_percent‚96.96717262268066$ disk./.usageGB‚589.8980178833008 disk.dm-1.out‚ 95.02734375 disk.drbd2165.out‚0 disk.rbd0.out‚0 cpu‚25.982588780400985 disk.sda1.out‚0 network.sent‚ 7.97929e+06# disk.nvme0c0n1.in‚ 5782.69140625 proc.cpu.threads‚17% disk.nvme0n1.out‚9167.62353515625 disk.sda1.in‚0 disk.dm-1.in‚0' proc.memory.availableMB‚ 496.8984375* proc.memory.percent‚16.995835304260254 network.recv‚ 1.169575e+06" disk.dm-0.out‚9167.62353515625 disk.sda2.out‚0 disk.rbd1.out‚11.3125 disk.sda.out‚ 95.02734375 disk.sda3.in‚0 disk.sda3.out‚ 95.02734375# proc.memory.rssMB‚ 2784.59765625 disk.sda2.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚39.58855093755284! disk.nvme0n1.in‚ 5782.69140625 disk.dm-0.in‚ 5782.69140625 disk.rbd1.in‚0fóX62É- _wandbruntime‚4349  _runtime‚4349Òz ÆÊÀ _runtime‚ 4349.85801859 _step‚408 & lr-AdamW/pg1‚0.0005064720162863987 & lr-AdamW/pg2‚0.0005064720162863987 trainer/global_step‚4999 % _timestamp‚1.7843670713520558e+09t‚C1ËËÅ _runtime‚ 4349.85801859 _step‚408 & lr-AdamW/pg1‚0.0005064720162863987 & lr-AdamW/pg2‚0.0005064720162863987 trainer/global_step‚4999 % _timestamp‚1.7843670713520558e+09˜újóé2Ì- _wandbruntime‚4349  _runtime‚4349®íåªâ'ÍÜO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764678711556608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46295871 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19515915 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39611852 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚760767460740233 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275216584075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46285894 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10755065 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711556608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5282939648512 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4999 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897746 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764679069935104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46285894 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069935104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167458716715008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172797739700224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329483 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46295871 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172797638802432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚409 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19516186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10754797 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843670713532007e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5338940699648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5274876034185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5283280987648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890ÿ.Í­ç'ÎáO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5338940699648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5274876034185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759395788947456 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5283280987648 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚764678711556608 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚759395771908096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46295871 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19515915 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39611852 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚760767460740233 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275216584075 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8761121 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46285894 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10755065 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚764678711556608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5282939648512 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4999 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85897746 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚764679069935104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46285894 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚764679069935104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167458716715008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172797739700224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚760767818329483 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46295871 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39611854 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚172797638802432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚409 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759395788947456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19516186 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10754797 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843670713532007e+09™.)©3“Ï‚(:ˆ ßíÒÒâàí gpu.0.gpu‚32.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚248.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W‚ ÂЂ(:· ßíÒ˜¼Ìð* proc.memory.percent‚16.995835304260254 cpu‚25.98385203407744 disk.drbd2165.out‚0 disk.dm-0.out‚ 9172.25390625$ memory_percent‚96.96881771087646 disk.dm-1.in‚0 network.sent‚ 7.997256e+06) disk./.usagePercent‚39.58855656971859 disk.sda1.in‚0 disk.rbd0.out‚0" disk.nvme0n1.out‚ 9172.25390625( proc.memory.availableMB‚ 496.62890625 disk.sda.out‚ 95.31640625 disk.sda1.out‚0# proc.memory.rssMB‚ 2784.59765625 disk.dm-1.out‚ 95.31640625 network.recv‚ 1.172834e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0! disk.nvme0n1.in‚ 5782.69140625 proc.cpu.threads‚17$ disk./.usageGB‚589.8981018066406 disk.dm-0.in‚ 5782.69140625# disk.nvme0c0n1.in‚ 5782.69140625 disk.sda3.out‚ 95.31640625$ disk.nvme0c0n1.out‚ 9172.25390625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚11.3125 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0Š¥¡2Ñ- _wandbruntime‚4350  _runtime‚4350Ÿ ¢ÛDÒæM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167492126861312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843670719891562e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39619484 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759543012937728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284293621760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284293621760 trainer/global_step‚4999 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764826965213184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39619484 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5276227637909 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8762863 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759543029977088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10757166 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755639095377920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275887087907 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19520032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85924423 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46304937 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚759543012937728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952275456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5339959944192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172832193987584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚410 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764827323598848 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10756899 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5340048514048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚760915340055189 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚4350.495295898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39619486 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19519762 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167492145473536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283952275456 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚755639112417280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_¤„ZÙ¨batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39619486 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762866 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46294946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154jA¼ñ&ÓëM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39619486 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19519762 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167492145473536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5283952275456 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚755639112417280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39619486 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8762866 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46294946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167492126861312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843670719891562e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39619484 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚759543012937728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5284293621760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5284293621760 trainer/global_step‚4999 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚764826965213184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39619484 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5276227637909 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8762863 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚759543029977088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46294946 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10757166 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755639095377920 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5275887087907 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19520032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85924423 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46304937 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚759543012937728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5283952275456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5339959944192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172832193987584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _step‚410 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚764827323598848 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10756899 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5340048514048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚760915340055189 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 _runtime‚4350.495295898 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0š< ×2Ô- _wandbruntime‚4350  _runtime‚4350Ï­¬­Õ‚ epoch‚12 trainer/global_step‚4999 $ _timestamp‚1.784367071990728e+09 _runtime‚4350.496411294 2loss/SRqqt1/assignment_loss‚0.5312351336040553 2loss/SRqqt2/assignment_loss‚0.5084292651060608 2loss/FBt1/assignment_loss‚0.025588059285558764 .loss/FBt2/detection_loss‚0.788581323887532 0loss/FRt1/assignment_loss‚1.9848729725326262 /loss/FRt1/detection_loss‚0.6055493225568361 /loss/FRt2/detection_loss‚1.4237749527469021 1loss/SRqqt2/detection_loss‚0.6998898212496593 _step‚411 0loss/FRt2/assignment_loss‚0.9128634239964172 3loss/FBt2/assignment_loss‚0.0032981292707396173 1loss/SRqqt1/detection_loss‚0.9061740532610899 .loss/FBt1/detection_loss‚0.475020408425301 %loss/total_loss‚8.865276865922777­ÊäP Ö‡ /loss/FRt2/detection_loss‚1.4237749527469021 1loss/SRqqt2/detection_loss‚0.6998898212496593 _step‚411 0loss/FRt2/assignment_loss‚0.9128634239964172 3loss/FBt2/assignment_loss‚0.0032981292707396173 1loss/SRqqt1/detection_loss‚0.9061740532610899 .loss/FBt1/detection_loss‚0.475020408425301 %loss/total_loss‚8.865276865922777 epoch‚12 trainer/global_step‚4999 $ _timestamp‚1.784367071990728e+09 _runtime‚4350.496411294 2loss/SRqqt1/assignment_loss‚0.5312351336040553 2loss/SRqqt2/assignment_loss‚0.5084292651060608 2loss/FBt1/assignment_loss‚0.025588059285558764 .loss/FBt2/detection_loss‚0.788581323887532 0loss/FRt1/assignment_loss‚1.9848729725326262 /loss/FRt1/detection_loss‚0.6055493225568361›v©~ׂ(:’ îíқ׶î gpu.0.gpu‚36.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áî˜ÖÁØ‚(:¶ îíÒãß´ï disk.rbd0.out‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚9178.9267578125 disk.sda3.out‚ 95.6171875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 5782.69140625 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚96.96788787841797) disk./.usagePercent‚39.58858754663024$ disk./.usageGB‚589.8985633850098 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚11.375 network.sent‚ 8.036385e+06 disk.sda1.out‚0 disk.dm-1.out‚ 95.6171875$ disk.nvme0n1.out‚9178.9267578125 disk.dm-0.in‚ 5782.69140625! disk.dm-0.out‚9178.9267578125# disk.nvme0c0n1.in‚ 5782.69140625 disk.sda.in‚0 network.recv‚ 1.177242e+06 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 95.6171875% proc.memory.availableMB‚ 496.78125# proc.memory.rssMB‚ 2784.59765625* proc.memory.percent‚16.995835304260254 cpu‚25.98517411592322å^O“Ù‚(:ˆ ýíÒ¦‘Ãî gpu.0.gpu‚57.0 gpu.0.memory‚20+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚277.134' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4268 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s^ByÉÚ‚(:¾ ýíÒýØäï disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 5884.59765625& disk.nvme0c0n1.out‚9181.6083984375 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 497.01953125 disk.sda2.in‚0 disk.sda3.in‚0$ disk.nvme0n1.out‚9181.6083984375 cpu‚25.986355111784327! disk.nvme0n1.in‚ 5884.59765625 disk.dm-0.in‚ 5884.59765625 disk.sda.out‚ 95.91796875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda3.out‚ 95.91796875 network.recv‚ 1.181573e+06* disk./.usagePercent‚39.588577562336404 disk.rbd0.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.995835304260254 proc.cpu.threads‚17 disk.dm-1.in‚0$ memory_percent‚96.96643352508545$ disk./.usageGB‚589.8984146118164! disk.dm-0.out‚9181.6083984375 disk.sda2.out‚0 disk.rbd1.out‚11.375# proc.memory.rssMB‚ 2784.59765625 disk.dm-1.out‚ 95.91796875 network.sent‚ 8.088704e+06âÙYå2Û- _wandbruntime‚4391  _runtime‚4391+½¬ÇÜÁ & lr-AdamW/pg1‚0.0005048974216615247 & lr-AdamW/pg2‚0.0005048974216615247 trainer/global_step‚5049 % _timestamp‚1.7843671128178432e+09 _runtime‚4391.324810255 _step‚412ø?&ÌÝÆ trainer/global_step‚5049 % _timestamp‚1.7843671128178432e+09 _runtime‚4391.324810255 _step‚412 & lr-AdamW/pg1‚0.0005048974216615247 & lr-AdamW/pg2‚0.0005048974216615247œ~@ò:2Þ- _wandbruntime‚4391  _runtime‚4391ćï$á'ßÛO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772091458540032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5333635150336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743777 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46750323 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚5049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5325492521093 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816918528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848371 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784367112818654e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10859220 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458540032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39993454 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39993452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46740346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174520270663168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169130154641408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8848368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766757823389696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _runtime‚4391.324910915 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5333635150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46750323 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5390025347584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5325833070983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚772091816918528 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚766757840429056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46740346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10858941 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766757840429056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768143968409991 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5333976489472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174520161376768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5333976489472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚762818135339008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39993452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768143610820741 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚413 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5390116021760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚762818118299648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19707309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Y0¹è%ààO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚766757840429056 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46740346 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169130136029184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10858941 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚766757840429056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚768143968409991 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5333976489472 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174520161376768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5333976489472 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚762818135339008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39993452 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚768143610820741 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚766757823389696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 _step‚413 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5390116021760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚762818118299648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 M@DeviceStatsMo iÁnitor.on_train_batch_start/inactive_split.all.freed‚19707309 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚772091458540032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5333635150336 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743777 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46750323 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚5049 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5325492521093 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚772091816918528 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8848371 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 $ _timestamp‚1.784367112818654e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10859220 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚772091458540032 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39993454 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39993452 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46740346 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚174520270663168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169130154641408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8848368 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚766757823389696 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _runtime‚4391.324910915 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5333635150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46750323 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5390025347584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5325833070983 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚772091816918528]ŽÎó2á- _wandbruntime‚4391  _runtime‚4391’Ü6ì&âæM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762964628971520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326503574815 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10860962 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762964646010880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚766905064419328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40001084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19711075 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5334647777280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326844124817 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86750482 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46749398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174554606287872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169163564787712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚772240070582272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5334989123584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8850116 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10861220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 trainer/global_step‚5049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5391041500160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760475 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174554715567104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚766905081458688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19711336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40001086 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚772239712196608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5334989123584 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8850116 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10861220 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 trainer/global_step‚5049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5391041500160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760475 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174554715567104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚766905081458688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19711336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40001086 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚772239712196608 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10860962 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚762964646010880 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚766905064419328 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40001084 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46759389 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19711075 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5334647777280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326844124817 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86750482 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46749398 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174554606287872 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚169163564787712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0žÁe,2ä- _wandbruntime‚4391  _runtime‚4391ü8ïí åƒ /loss/FRt1/detection_loss‚0.5385401794062854 /loss/FRt2/detection_loss‚1.5343313704912172 %loss/total_loss‚8.985509518215794 trainer/global_step‚5049 /loss/FRt1/assignment_loss‚2.117672198118847 3loss/FBt1/assignment_loss‚0.0023380920076509636 1loss/SRqqt2/detection_loss‚0.6637366832441076 /loss/FBt2/detection_loss‚0.8413394302858607 /loss/FRt2/assignment_loss‚0.860049962731937 2loss/SRqqt1/assignment_loss‚0.6687269222902907 2loss/SRqqt2/assignment_loss‚0.4907402299399989 1loss/FBt2/assignment_loss‚0.06345393913619175 1loss/SRqqt1/detection_loss‚0.8858258716818813 0loss/FBt1/detection_loss‚0.31875463888152605 epoch‚12 _runtime‚4391.955913995 % _timestamp‚1.7843671134502366e+09 _step‚415˜ëÆâæˆ 2loss/SRqqt1/assignment_loss‚0.6687269222902907 2loss/SRqqt2/assignment_loss‚0.4907402299399989 1loss/FBt2/assignment_loss‚0.06345393913619175 1loss/SRqqt1/detection_loss‚0.8858258716818813 0loss/FBt1/detection_loss‚0.31875463888152605 epoch‚12 _runtime‚4391.955913995 % _timestamp‚1.7843671134502366e+09 _step‚415 /loss/FRt1/detection_loss‚0.5385401794062854 /loss/FRt2/detection_loss‚1.5343313704912172 %loss/total_loss‚8.985509518215794 trainer/global_step‚5049 /loss/FRt1/assignment_loss‚2.117672198118847 3loss/FBt1/assignment_loss‚0.0023380920076509636 1loss/SRqqt2/detection_loss‚0.6637366832441076 /loss/FBt2/detection_loss‚0.8413394302858607 /loss/FRt2/assignment_loss‚0.860049962731937Ÿõs^+žç‚(:“ ŒíÒ¬¨…î gpu.0.gpu‚40.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L‘wþÄè‚(:¹ ŒíÒòþûï disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 96.3359375 disk.sda1.out‚0* proc.memory.percent‚16.995835304260254 network.sent‚ 8.093536e+06 network.recv‚ 1.184697e+06 disk.rbd0.out‚0" disk.dm-0.out‚9186.06591796875* disk./.usagePercent‚39.588581914464484 disk.dm-1.out‚ 96.3359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 5884.59765625 disk.rbd1.in‚0 disk.rbd1.out‚11.4375 disk.sda.in‚0! disk.nvme0n1.in‚ 5884.59765625% disk.nvme0n1.out‚9186.06591796875' disk.nvme0c0n1.out‚9186.06591796875 disk.sda.out‚ 96.3359375$ memory_percent‚96.96807861328125" proc.memory.availableMB‚496.75# proc.memory.rssMB‚ 2784.59765625 proc.cpu.threads‚17$ disk./.usageGB‚589.8984794616699 disk.dm-1.in‚0 cpu‚25.987530502503922 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 5884.59765625 disk.sda1.in‚09µ1ƒ›é‚(: ›íÒ»Çàí gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë}û·Ãê‚(:¸ ›íÒÅ£´ï disk.rbd0.in‚0! disk.nvme0n1.in‚ 5884.59765625 disk.dm-0.out‚9189.193359375 disk.sda2.in‚0# proc.memory.rssMB‚ 2784.59765625$ disk./.usageGB‚589.8983993530273 disk.rbd1.out‚11.4375 disk.sda3.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0# disk.nvme0n1.out‚9189.193359375# disk.nvme0c0n1.in‚ 5884.59765625 disk.sda.in‚0 disk.sda.out‚ 96.6640625 network.sent‚ 8.156509e+06 disk.dm-0.in‚ 5884.59765625$ memory_percent‚96.96815013885498% disk.nvme0c0n1.out‚9189.193359375 disk.sda1.in‚0 disk.dm-1.in‚0 cpu‚25.988932458367795 disk.sda1.out‚0) disk./.usagePercent‚39.58857653830626 disk.rbd1.in‚0 disk.dm-1.out‚ 96.6640625 disk.drbd2165.out‚0 disk.sda3.out‚ 96.6640625( proc.memory.availableMB‚ 496.73828125* proc.memory.percent‚16.995835304260254 network.recv‚ 1.189393e+06 disk.drbd2165.in‚0½*[Áœë‚(:‘ ªíÒˆ¶Òí gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߶ì‚(:« ªíÒ¦õºï disk.dm-0.out‚ 9194.40234375 disk.sda1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 1.19291e+06! disk.nvme0n1.in‚ 5884.59765625# disk.nvme0c0n1.in‚ 5884.59765625$ disk.nvme0c0n1.out‚ 9194.40234375 disk.sda.in‚0 disk.sda3.in‚0 cpu‚25.99014901146907 disk.rbd1.out‚11.5 disk.sda.out‚ 96.89453125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.995835304260254 disk.dm-0.in‚ 5884.59765625 disk.sda2.in‚0 disk.dm-1.out‚ 96.89453125 disk.drbd2165.in‚0 disk./.usageGB‚ 589.8984375" disk.nvme0n1.out‚ 9194.40234375 disk.dm-1.in‚0( disk./.usagePercent‚39.5885790983816 disk.sda3.out‚ 96.89453125# proc.memory.rssMB‚ 2784.59765625 network.sent‚ 8.16145e+06 disk.sda2.out‚0 disk.rbd1.in‚0$ memory_percent‚96.96824550628662( proc.memory.availableMB‚ 496.72265625(ŠU}2í- _wandbruntime‚4433  _runtime‚4433ÕÍÀÐÇîÁ & lr-AdamW/pg1‚0.0005033090895343296 & lr-AdamW/pg2‚0.0005033090895343296 trainer/global_step‚5099 % _timestamp‚1.7843671545238194e+09 _runtime‚4433.029802146 _step‚416[R…ÌïÆ _runtime‚4433.029802146 _step‚416 & lr-AdamW/pg1‚0.0005033090895343296 & lr-AdamW/pg2‚0.0005033090895343296 trainer/global_step‚5099 % _timestamp‚1.7843671545238194e+09 ×¢¦›2ð- _wandbruntime‚4433  _runtime‚44337(ø>`ñÜO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774119874871296 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5384671991296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚779504563901952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10962798 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205523456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚774119874871296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170801592567808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceSta­S±Û‚ tsMonitor.on_train_batch_start/active.large_pool.allocated‚40375054 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5384671991296 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚779504563901952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10962542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557891 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40375052 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775519760901249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚5099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204775 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774119891910656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19898160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8935621 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5441193600512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47194798 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204775 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843671545249481e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5384330652160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330652160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770143668932608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176242683173376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚770143651893248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40375054 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8935618 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775520118490499 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87569850 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170801573955584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4433.034729992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5441109217792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47194798 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19898419 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40375052 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269B7!nç'òáO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5384671991296 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚779504563901952 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10962798 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚779504205523456 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚774119874871296 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170801592567808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40375054 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5384671991296 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚779504563901952 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚774119891910656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10962542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5376449557891 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40375052 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775519760901249 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 trainer/global_step‚5099 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47204775 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚774119891910656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19898160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8935621 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5441193600512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47194798 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47204775 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7843671545249481e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5384330652160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5384330652160 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770143668932608 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176242683173376 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚770143651893248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40375054 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8935618 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775520118490499 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87569850 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170801573955584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4433.034729992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5441109217792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47194798 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19898419 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40375052 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚774119874871296¡£Õ?Ñ2ó- _wandbruntime‚4433  _runtime‚4433#¸ÚLì&ôæM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47213841 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684625408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚770290162565120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10964875 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚4433.666112191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40382684 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779652459180032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176277127625728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170835002714112 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5377460611725 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779652817565696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19902241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203850 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774267115900928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19901976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5385684625408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774267132940288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5377120061723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47203850 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5385343279104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚770290179604480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚5099 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ßÀèËVõëM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚770290162565120 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10964875 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 _runtime‚4433.666112191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40382684 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚779652459180032 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176277127625728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170835002714112 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586534 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5377460611725 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚779652817565696 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19902241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47203850 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚774267115900928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19901976 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5385684625408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚774267132940288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5377120061723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 NADe™ÓSn›viceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47203850 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5385343279104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚770290179604480 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 trainer/global_step‚5099 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47213841 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5385684625408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405902 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚781481926352896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5435026153984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782896268571007 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5435026153984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40756652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20088694 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177965368992256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952506880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11065826 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044799 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786917310885376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786916952506880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843671959796615e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981757 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5492338498048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5492258309632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚4474.485889712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786917310885376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20088948 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172473030494208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9022868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5426725494909 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11066077 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47649250 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5149 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9022871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367493120 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177965270191616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚421 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172473011881984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Ò‹kæ'„àO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786917310885376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786916952506880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚777469185486848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 % _timestamp‚1.7843671959796615e+09 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚782895910981757 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5492338498048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5492258309632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚4474.485889712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚786917310885376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20088948 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172473030494208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚781481943392256 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40756654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9022868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5426725494909 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47659227 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11066077 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47649250 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5149 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9022871 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435367493120 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177965270191616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 _step‚421 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172473011881984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40756654 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚781481943392256 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435367493120 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚777469202526208 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88405902 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚781481926352896 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5435026153984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚782896268571007 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5435026153984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40756652 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20088694 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177965368992256 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786916952506880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11065826 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5427066044799¥97´2…- _wandbruntime‚4475  _runtime‚4475H(ýÿ‡†åM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88432579 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47668293 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚781629167382528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707351 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40764286 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚787065564549120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40764284 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚777615696158720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚781629184421888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11068100 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚787065564549120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843671966133573e+09 _step‚422 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚781629167382528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9024616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20092453 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5436380127232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47668293 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177999827736576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47658302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚783043790296713 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172506440640512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5493288302592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5436380127232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚787065206163456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47658302 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5427736548631 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4475.119490934 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177999728943104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40764286 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098633 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11067840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438Ø"ð&‡êM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11067840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88432579 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚5149 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47668293 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚781629167382528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚98793472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚783043432707351 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40764286 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚80181248 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚787065564549120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40764284 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚777615696158720 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚781629184421888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11068100 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚787065564549120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843671966133573e+09 _step‚422 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚781629167382528 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9024616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20092453 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚781629184421888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5436380127232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47668293 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9024613 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚177999827736576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47658302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚783043790296713 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚172506440640512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5493288302592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5436380127232 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚787065206163456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47658302 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5427736548631 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚4475.119490934 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40764284 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177999728943104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40764286 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428077098633 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0¦»Ãøa2ˆ- _wandbruntime‚4475  _runtime‚4475p-Ÿ‰€ 2loss/SRqqt1/assignment_loss‚0.6609026602741324 2loss/SRqqt2/assignment_loss‚0.5963549976348844 /loss/FBt2/detection_loss‚1.0974732733198576 % _timestamp‚1.7843671966149952e+09 0loss/FRt2/assignment_loss‚0.9086652927058723 /loss/FRt2/detection_loss‚1.3318978194269662 1loss/SRqqt1/detection_loss‚1.0399318264317117 /loss/FRt1/detection_loss‚0.6027803578460933 %loss/total_loss‚9.239625712367278 _runtime‚4475.120684689 .loss/FRt1/assignment_loss‚1.94189944048846 2loss/FBt1/assignment_loss‚0.037143604501634446 1loss/FBt2/assignment_loss‚0.07529291938400218 0loss/SRqqt2/detection_loss‚0.629629405588034 /loss/FBt1/detection_loss‚0.3176541147656294 epoch‚12 trainer/global_step‚5149 _step‚423ðþ Š… .loss/FRt1/assignment_loss‚1.94189944048846 2loss/FBt1/assignment_loss‚0.037143604501634446 1loss/FBt2/assignment_loss‚0.07529291938400218 0loss/SRqqt2/detection_loss‚0.629629405588034 /loss/FBt1/detection_loss‚0.3176541147656294 epoch‚12 trainer/global_step‚5149 _step‚423 2loss/SRqqt1/assignment_loss‚0.6609026602741324 2loss/SRqqt2/assignment_loss‚0.5963549976348844 /loss/FBt2/detection_loss‚1.0974732733198576 % _timestamp‚1.7843671966149952e+09 0loss/FRt2/assignment_loss‚0.9086652927058723 /loss/FRt2/detection_loss‚1.3318978194269662 1loss/SRqqt1/detection_loss‚1.0399318264317117 /loss/FRt1/detection_loss‚0.6027803578460933 %loss/total_loss‚9.239625712367278 _runtime‚4475.120684689§¿…Õ0“‹‚(:ˆ æíÒÒ¥ªí gpu.0.gpu‚94.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚254.517' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.9034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_JH(ËŒ‚(:À æíÒã‘ð disk.rbd1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 97.96484375# proc.memory.rssMB‚ 2784.59765625# disk.nvme0c0n1.in‚ 6001.72265625 disk.rbd1.out‚11.625' disk.nvme0c0n1.out‚9209.79052734375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 6001.72265625" disk.dm-0.out‚9209.79052734375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.out‚ 97.96484375 disk.sda2.out‚0 network.recv‚ 1.207525e+06 disk.sda1.in‚0! disk.nvme0n1.in‚ 6001.72265625$ disk./.usageGB‚589.8983383178711 disk.drbd2165.in‚0% disk.nvme0n1.out‚9209.79052734375' proc.memory.availableMB‚ 496.8203125 disk.sda.out‚ 97.96484375 disk.sda3.in‚0 disk.sda2.in‚0 cpu‚25.995039234355453 network.sent‚ 8.239314e+06* disk./.usagePercent‚39.588572442185715 disk.rbd0.out‚0$ memory_percent‚96.96764945983887* proc.memory.percent‚16.995835304260254¬Q˜‘‚(:† õíÒ¾¦¨î gpu.0.gpu‚92.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôøW¿Ž‚(:´ õíÒ´Òï disk.rbd1.out‚11.625' disk.nvme0c0n1.out‚9212.90283203125# proc.memory.rssMB‚ 2784.59765625 disk.dm-0.in‚ 6109.5546875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 98.234375 network.sent‚ 8.302526e+06) disk./.usagePercent‚39.58856117785421$ disk./.usageGB‚589.8981704711914 disk.rbd0.in‚0" disk.dm-0.out‚9212.90283203125$ memory_percent‚96.96760177612305* proc.memory.percent‚16.995835304260254 disk.sda.in‚0 disk.sda3.out‚ 98.234375& proc.memory.availableMB‚ 496.828125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6109.5546875 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 6109.5546875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 network.recv‚ 1.21227e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 98.234375 disk.drbd2165.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚9212.90283203125 cpu‚25.996228403935458å`Ü$ž‚(:“ „‘íÒ˜œšï gpu.0.gpu‚86.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚247.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.404199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú¥:œÆ‚(:» „‘íÒýÐ¥ð% disk.nvme0n1.out‚9217.83349609375$ memory_percent‚96.96757793426514( proc.memory.availableMB‚ 496.83203125 network.sent‚ 8.309505e+06) disk./.usagePercent‚39.58855170557544$ disk./.usageGB‚589.8980293273926 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 6109.5546875 disk.dm-1.out‚ 98.4140625 cpu‚25.997383141601574 disk.sda.in‚0 disk.dm-0.in‚ 6109.5546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚11.6875 disk.sda3.out‚ 98.4140625 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2784.59765625' disk.nvme0c0n1.out‚9217.83349609375 disk.sda.out‚ 98.4140625 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.dm-0.out‚9217.83349609375 disk.sda1.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 6109.5546875* proc.memory.percent‚16.995835304260254 network.recv‚ 1.215761e+06 disk.sda3.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 íÕW2‘- _wandbruntime‚4516  _runtime‚4516}Q UÆ’À _runtime‚4516.161839254 _step‚424 & lr-AdamW/pg1‚0.0005000916217111753 & lr-AdamW/pg2‚0.0005000916217111753 trainer/global_step‚5199 $ _timestamp‚1.784367237654716e+09ÍŽ_¨Ë“Å _step‚424 & lr-AdamW/pg1‚0.0005000916217111753 & lr-AdamW/pg2‚0.0005000916217111753 trainer/global_step‚5199 $ _timestamp‚1.784367237654716e+09 _runtime‚4516.161839254¨t~ˆ2”- _wandbruntime‚4516  _runtime‚4516Œ“f—ò•ÜO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5486062994944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41138254 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41138252 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚790272061062265 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48113679 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚784794736119808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20280181 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788843977834496 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5543494411776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41138254 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174144468420608 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843672376554563e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚794329699490304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699490304 C:DeviceStatsMonitor.on_tž)—òð rain_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚788843994873856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48103702 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4516.161965532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚794330057868800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5485721655808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48103702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5485721655808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚790272418651515 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9110118 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794329699490304 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚788843994873856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48103702 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4516.161965532 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚794330057868800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5485721655808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48103702 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5485721655808 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚790272418651515 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9110118 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11172138 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11171870 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5544417940480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚426 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41145886 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784941246791680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179722409980928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20283733 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚794478311532544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258638 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9111863 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953146880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0T&‘kè"™ëM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚788991235903488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41145884 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚794478311532544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5478693585541 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 % _timestamp‚1.7843672382852876e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚788991218864128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚784941229752320 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚784941246791680 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179722409980928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚94861312 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788991218864128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20283733 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚794478311532544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89258638 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9111863 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚794477953146880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5487075629056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41145884 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚89258638 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5487075629056 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48112754 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41145886 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 _runtime‚4516.791379369 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5486734282752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5478353035539 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268631 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174177878566912 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 113473536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788991235903488 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5486734282752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11172138 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11171870 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5544417940480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 _step‚426 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41145886 E&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6161645650863647 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32880184054374695 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5864877700805664 jet/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5493547916412354 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5284069776535034 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2688369154930115 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2378387451171875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23809519410133362 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 % _timestamp‚1.7843672562823117e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22793300449848175 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904025197029114 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3922165334224701 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996178925037384 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9995894432067871 )loss/val_total_loss‚9.133208494003421 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11987075209617615 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2535034418106079 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2331872433423996 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2153335064649582 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 &particle/f_score‚0.620334804058075 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9904516935348511 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48070475459098816 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6867733001708984 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2258526086807251 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08943793922662735 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚5199 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8458911180496216 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1452999711036682 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3312656283378601 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6672690510749817 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.4657842218875885 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1452999711036682 particle/accuracy_4_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 *jet/accuracy_1_of_4‚0.9429579377174377 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8735461235046387 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6322776675224304 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2760416865348816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_3‚0.9063854217529297 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5923033952713013 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5347026586532593 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017110243439674377 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5554150342941284 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49473676085472107 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22643211483955383 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2393638789653778 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.99079430103302 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17027908563613892 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6422579288482666 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5579546689987183 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2258526086807251 +jet/accuracy_2_of_2‚0.12645471096038818 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.30324074625968933 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7303362488746643 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0479423813521862 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6129752397537231 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1504824012517929 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5806663036346436 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.588641881942749 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5261483788490295 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2654375731945038 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22231093049049377 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971383213996887 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895139336585999 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7664399147033691 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8414967060089111 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6390718221664429 +particle/specificity‚0.9097942113876343 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23831793665885925 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5717512965202332 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15100349485874176 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_2‚0.7590689659118652 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9966822862625122 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.503991961479187 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2957310378551483 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6362266540527344 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5774327516555786 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8479025959968567 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.029750943183898926 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5611549615859985 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5579546689987183 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_2_of_4‚0.7307325005531311 jet/accuracy_3_of_5‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6399547457695007 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8434996008872986 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6425186991691589 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958134651184 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.024862440302968025 7validation_average_jet_accuracy‚0.34418758749961853 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 _step‚428 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2770431637763977 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25798743963241577 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49584898352622986 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6588315367698669 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6358707547187805 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9954329133033752 +particle/sensitivity‚0.5979479551315308 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6356087327003479 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.553069531917572 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2876436412334442 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19038984179496765 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2544991970062256 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.594109833240509 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12224012613296509 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9790278673171997 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968152046203613 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.492718368768692 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03290402144193649 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2388603836297989 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8359978199005127 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.204420268535614 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1æ‹p2¡ùz 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9008520841598511 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18087774515151978 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22679579257965088 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8428237438201904 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9982993602752686 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19732797145843506 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6161645650863647 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32880184054374695 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5864877700805664 jet/accuracy_2_of_5‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5493547916412354 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5284069776535034 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2688369154930115 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2378387451171875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23809519410133362 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5491666793823242 % _timestamp‚1.7843672562823117e+09 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22793300449848175 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904025197029114 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3922165334224701 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996178925037384 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_3_of_3‚0.9995894432067871 )loss/val_total_loss‚9.133208494003421 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11987075209617615 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2535034418106079 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2331872433423996 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2153335064649582 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 &particle/f_score‚0.620334804058075 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9904516935348511 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48070475459098816 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6867733001708984 particle/accuracy_3_of_5‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2258526086807251 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.08943793922662735 ("Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚5199 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8458911180496216 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1452999711036682 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3312656283378601 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6672690510749817 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.4657842218875885 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1452999711036682 particle/accuracy_4_of_5‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8425925374031067 *jet/accuracy_1_of_4‚0.9429579377174377 ("Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8735461235046387 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6322776675224304 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2760416865348816 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_3‚0.9063854217529297 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5923033952713013 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5347026586532593 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017110243439674377 particle/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5554150342941284 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49473676085472107 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22643211483955383 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2393638789653778 5 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.99079430103302 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17027908563613892 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6422579288482666 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5579546689987183 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2258526086807251 +jet/accuracy_2_of_2‚0.12645471096038818 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.30324074625968933 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7303362488746643 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0479423813521862 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6129752397537231 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12333333492279053 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1504824012517929 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5806663036346436 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.588641881942749 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5261483788490295 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2654375731945038 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22231093049049377 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9971383213996887 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9895139336585999 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7664399147033691 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8414967060089111 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6390718221664429 +particle/specificity‚0.9097942113876343 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23831793665885925 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5717512965202332 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15100349485874176 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_2‚0.7590689659118652 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9966822862625122 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.503991961479187 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2957310378551483 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976394176483154 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6362266540527344 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5774327516555786 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8479025959968567 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.029750943183898926 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5611549615859985 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5579546689987183 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_2_of_4‚0.7307325005531311 jet/accuracy_3_of_5‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6399547457695007 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8434996008872986 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6425186991691589 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958134651184 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.024862440302968025 7validation_average_jet_accuracy‚0.34418758749961853 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 _step‚428 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 & Purity/0FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2770431637763977 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25798743963241577 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49584898352622986 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6588315367698669 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6358707547187805 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9954329133033752 +particle/sensitivity‚0.5979479551315308 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6356087327003479 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.553069531917572 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2876436412334442 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19038984179496765 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_4‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2544991970062256 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.594109833240509 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12224012613296509 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9790278673171997 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9968152046203613 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.492718368768692 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03290402144193649 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2388603836297989 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8359978199005127 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.204420268535614 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999195098876951Ø‘ 31 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.921983540058136 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6500509977340698 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6496788263320923 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6867733001708984 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15211956202983856 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 /particle/accuracy_3_of_4‚0.9936507940292358 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8558579683303833 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5927037000656128 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.552936315536499 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9987881779670715 +jet/accuracy_1_of_1‚0.30224674940109253 particle/accuracy_1_of_2‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.02777777798473835 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 .particle/accuracy_4_of_4‚0.937131404876709 _runtime‚4534.788899274 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21749112010002136 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5929052233695984 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1530439257621765 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.02777777798473835 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9341243505477905 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_3_of_3‚0.06898641586303711 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03996586054563522 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8425925374031067 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2117644101381302 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9861111044883728 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6633256077766418 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5901408195495605 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 epoch‚12 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5379846692085266 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2578125 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2146693915128708 jet/accuracy_4_of_5‚0¬જѓ¢‚(:ˆ ¢‘íÒÆîï gpu.0.gpu‚84.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NþãÊã‚(:¸ ¢‘íÒæωò$ memory_percent‚97.04058170318604" proc.memory.rssMB‚ 2797.0078125 network.sent‚ 8.375131e+06* disk./.usagePercent‚39.588568090057635 disk.rbd0.in‚0& disk.nvme0c0n1.out‚9229.2333984375 disk.sda.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.07158088684082$ disk./.usageGB‚589.8982734680176 disk.sda3.out‚ 99.17578125 disk.dm-1.in‚0 disk.dm-0.in‚ 6109.5546875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 6109.5546875 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 99.17578125 disk.drbd2165.in‚0! disk.dm-0.out‚9229.2333984375 proc.cpu.threads‚17$ disk.nvme0n1.out‚9229.2333984375 cpu‚26.012915976738206 network.recv‚ 1.223089e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 6109.5546875( proc.memory.availableMB‚ 484.87109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚11.75 disk.sda.out‚ 99.17578125 disk.sda1.in‚0·U“¤‚(:ˆ ±‘íÒ…—Èí gpu.0.gpu‚83.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BAÌ’Ä¥‚(:¹ ±‘íÒÜÁÄñ disk.sda2.out‚0! disk.nvme0n1.in‚ 6202.10546875$ memory_percent‚97.04296588897705" proc.memory.rssMB‚ 2797.0078125 disk.dm-0.in‚ 6202.10546875& disk.nvme0c0n1.out‚9233.0751953125 disk.rbd0.in‚0( proc.memory.availableMB‚ 484.48046875 cpu‚26.01397066474779 disk.sda2.in‚0 disk.sda3.out‚ 99.37890625$ disk.nvme0n1.out‚9233.0751953125 network.sent‚ 8.433423e+06) disk./.usagePercent‚39.58855298561311 disk.rbd1.out‚11.75 disk.dm-1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.227321e+06! disk.dm-0.out‚9233.0751953125# disk.nvme0c0n1.in‚ 6202.10546875) proc.memory.percent‚17.07158088684082 disk.sda.out‚ 99.37890625 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 99.37890625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚589.8980484008789 disk.sda.in‚0Óz3,“¦‚(:ˆ À‘íҨ߀ï gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚34.0 gpu.0.powerWatts‚108.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7634 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆŒ³m½§‚(:² À‘íÒñªñ' proc.memory.availableMB‚ 484.3203125! disk.dm-0.out‚9239.3740234375 disk.sda3.out‚99.65625$ disk.nvme0n1.out‚9239.3740234375 proc.cpu.threads‚17 network.sent‚ 8.436387e+06 disk.sda2.in‚0 disk.rbd1.out‚11.8125& disk.nvme0c0n1.out‚9239.3740234375 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sda.out‚99.65625 disk.sda1.out‚0$ disk./.usageGB‚589.8981475830078 disk.dm-1.out‚99.65625 disk.rbd0.out‚0" proc.memory.rssMB‚ 2797.0078125) proc.memory.percent‚17.07158088684082 cpu‚26.01490958202578 disk.sda2.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 6202.10546875$ memory_percent‚97.04394340515137 disk.dm-0.in‚ 6202.10546875 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0 network.recv‚ 1.230523e+06* disk./.usagePercent‚39.588559641809006# disk.nvme0c0n1.in‚ 6202.10546875÷s2¨- _wandbruntime‚4575  _runtime‚4575+ËÅ©¿ % _timestamp‚1.7843672973459325e+09 _runtime‚4575.851887003 _step‚429 % lr-AdamW/pg1‚0.000498462692667162 % lr-AdamW/pg2‚0.000498462692667162 trainer/global_step‚5249–z ¯ÊªÄ % lr-AdamW/pg1‚0.000498462692667162 % lr-AdamW/pg2‚0.000498462692667162 trainer/global_step‚5249 % _timestamp‚1.7843672973459325e+09 _runtime‚4575.851887003 _step‚429­ƒ…×2«- _wandbruntime‚4575  _runtime‚4575d ‚3â'¬ÜO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545678193954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11305286 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5612707863040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181444507697664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚794028932927488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746329600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5554150363136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746329600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181444619081216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚799574594082082 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843672973469963e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41640518 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48638453 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104708096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360=iç'­áO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48628476 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5554491702272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5546018743844 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚430 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚798135613005824 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41640520 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11305018 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5554150363136 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803689746329600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181444619081216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚798135613005824 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚799574594082082 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843672973469963e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41640518 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48638453 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚803690104708096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41640518 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5545678193954 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11305286 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5612707863040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚181444507697664 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚794028932927488 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803689746329600 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚798135595966464 B2®- _wandbruntime‚4576  _runtime‚4576Á™…¯åM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285678 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285678 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚277 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48647519 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚798282854035456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚803837999986176 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295671 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48637528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚799722115807676 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504336384 _step‚431 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5555162990080 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784367297978813e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚798282836996096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798282836996096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41648150 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647519 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20508921 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚799722473397038 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚794175443599360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41648152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181478967798784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_¾_ÖNend/inactive_split.all.peak‚541 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚798282854035456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚4576.484919439 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9201636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11307285 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11307011 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797678 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20508644 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5613739205632 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9201633 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20508644 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5613739205632 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9201633 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48637528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚175865247205376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚799722115807676 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5555504336384 _step‚431 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5555162990080 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 $ _timestamp‚1.784367297978813e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚798282836996096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚798282836996096 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41648150 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48647519 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20508921 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚799722473397038 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚794175443599360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41648152 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚181478967798784 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚798282854035456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41648152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 _runtime‚4576.484919439 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 100890624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚82278400 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9201636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11307285 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11307011 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5547029797678 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805497664487424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11408723 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11408456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177503256373248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚183167130342912 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811102851691520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49092905 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5604845864960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801354449481728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493313024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805497647448064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187204096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 _runtime‚4617.352226741 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚811102493313024 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806950744162590 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49082928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5604845864960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801354466521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187204096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843673388461018e+09 _step‚434 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5663987450368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚806950744162590 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42022118 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49082928 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5604845864960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚805497664487424 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801354466521088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5605187204096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 % _timestamp‚1.7843673388461018e+09 _step‚434 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 UGDeviÉ3FàiceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5663987450368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚805497664487424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11408723 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11408456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚177503256373248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚183167130342912 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚811102851691520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49092905 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5604845864960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801354449481728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚811102493313024 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚805497647448064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5299 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5605187204096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376²›ç/2¾- _wandbruntime‚4617  _runtime‚4617Íͺ÷ë&¿åM A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚811250746969600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚805644905517056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20699317 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9288886 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49101971 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49101971 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177536685131776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11410710 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚4617.986639228 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199838208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20699596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183201590548480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5299 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121730 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131723 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚811251105355264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42029752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183201695633408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5605858491904 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚811251105355264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177536666519552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚805644888477696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5664924028928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029750 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5606199838208 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9288883 _step‚435 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121730 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5597305734584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ì6‰ôð&ÀêM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9288883 _step‚435 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121730 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5597305734584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚811250746969600 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚805644905517056 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20699317 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9288886 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49101971 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49101971 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177536685131776 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11410710 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 _runtime‚4617.986639228 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5606199838208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20699596 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚183201590548480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 trainer/global_step‚5299 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121730 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131723 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚811251105355264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42029752 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚183201695633408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5605858491904 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚811251105355264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177536666519552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚805644888477696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5664924028928 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42029750 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5606199838208 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438³RXé2Á- _wandbruntime‚4617  _runtime‚4617Ï!|´ ƒ 2loss/SRqqt1/assignment_loss‚0.5366922838528269 /loss/FBt1/detection_loss‚0.4058519921618803 %loss/total_loss‚8.436306666868834 epoch‚13 % _timestamp‚1.7843673394819348e+09 _runtime‚4617.987946906 2loss/FBt2/assignment_loss‚0.019247304274004617 /loss/FRt2/detection_loss‚1.4520554642678043 1loss/SRqqt1/detection_loss‚0.9217657353759916 0loss/FRt2/assignment_loss‚0.7648833829003412 1loss/FBt1/assignment_loss‚0.01631404573854466 /loss/FRt1/detection_loss‚0.5790090395280273 1loss/SRqqt2/detection_loss‚0.6411054145467939 /loss/FBt2/detection_loss‚0.7784765826968046 _step‚436 0loss/FRt1/assignment_loss‚1.8119909525281046 2loss/SRqqt2/assignment_loss‚0.5089144689977111 trainer/global_step‚5299»2ñNÈ 2loss/SRqqt1/assignment_loss‚0.5366922838528269 /loss/FBt1/detection_loss‚0.4058519921618803 %loss/total_loss‚8.436306666868834 epoch‚13 % _timestamp‚1.7843673394819348e+09 _runtime‚4617.987946906 2loss/FBt2/assignment_loss‚0.019247304274004617 /loss/FRt2/detection_loss‚1.4520554642678043 1loss/SRqqt1/detection_loss‚0.9217657353759916 0loss/FRt2/assignment_loss‚0.7648833829003412 1loss/FBt1/assignment_loss‚0.01631404573854466 /loss/FRt1/detection_loss‚0.5790090395280273 1loss/SRqqt2/detection_loss‚0.6411054145467939 /loss/FBt2/detection_loss‚0.7784765826968046 _step‚436 0loss/FRt1/assignment_loss‚1.8119909525281046 2loss/SRqqt2/assignment_loss‚0.5089144689977111 trainer/global_step‚5299´~`užÄ‚(:“ í‘íÒü‰ÿí gpu.0.gpu‚67.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ìZÇÃÅ‚(:¸ í‘íÒ–¡öñ disk.dm-1.out‚ 100.82421875 disk.sda2.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 6306.921875 disk.sda1.out‚0" proc.memory.rssMB‚ 2797.0078125 disk.sda3.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 6306.921875 disk.rbd1.in‚0 disk.rbd1.out‚11.875 network.recv‚ 1.241879e+06 disk.sda.out‚ 100.82421875$ memory_percent‚97.04418182373047 proc.cpu.threads‚17 network.sent‚ 8.522202e+06 disk.rbd0.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.07158088684082) disk./.usagePercent‚39.58859292278846$ disk./.usageGB‚589.8986434936523 disk.sda3.out‚ 100.82421875 disk.dm-1.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚9250.75927734375 disk.sda1.in‚0% proc.memory.availableMB‚ 484.28125% disk.nvme0n1.out‚9250.75927734375" disk.dm-0.out‚9250.75927734375! disk.nvme0c0n1.in‚ 6306.921875 cpu‚26.017930038416463>Á¶‡Æ‚(:… ü‘íÒœ‰Ýï gpu.0.gpu‚21.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O≵ºÇ‚(:¯ ü‘íÒØŠüñ cpu‚26.01905209005534 proc.cpu.threads‚17 network.recv‚ 1.246568e+06 disk.sda3.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.07158088684082) disk./.usagePercent‚39.58858908267544 disk.dm-1.in‚0 disk.drbd2165.in‚0# memory_percent‚97.0463514328003 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6306.921875! disk.dm-0.out‚9256.5869140625 disk.rbd1.out‚12$ disk./.usageGB‚589.8985862731934 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 8.574072e+06$ disk.nvme0n1.out‚9256.5869140625! disk.nvme0c0n1.in‚ 6306.921875 disk.sda3.out‚ 101.2578125 disk.sda2.out‚0 disk.sda.out‚ 101.2578125( proc.memory.availableMB‚ 483.92578125 disk.dm-1.out‚ 101.2578125 disk.dm-0.in‚ 6306.921875 disk.sda.in‚0" proc.memory.rssMB‚ 2797.0078125 disk.rbd1.in‚0& disk.nvme0c0n1.out‚9256.5869140625qÜ8²žÈ‚(:“ ‹’íÒ™³í gpu.0.gpu‚92.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬“½žÉ‚(:² ‹’íÒñà·ï disk.dm-1.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚9261.0302734375 disk.dm-0.in‚ 6306.921875$ memory_percent‚97ïKÌ{ .04484939575195 disk.sda.in‚0" proc.memory.rssMB‚ 2797.0078125$ disk./.usageGB‚589.8986854553223 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 6306.921875 disk.sda2.in‚0 network.sent‚ 8.576994e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚9261.0302734375 disk.sda.out‚ 101.69921875 disk.sda1.out‚0) proc.memory.percent‚17.07158088684082 cpu‚26.019983115096014 proc.cpu.threads‚17 network.recv‚ 1.249965e+06) disk./.usagePercent‚39.58859573887133 disk.dm-1.out‚ 101.69921875! disk.dm-0.out‚9261.0302734375 disk.sda2.out‚0& proc.memory.availableMB‚ 484.171875 disk.sda3.out‚ 101.69921875! disk.nvme0c0n1.in‚ 6306.921875 disk.sda1.in‚0 disk.rbd1.out‚12ê䓘2Ê- _wandbruntime‚4658  _runtime‚4658xn“UÇËÁ trainer/global_step‚5349 % _timestamp‚1.7843673804459164e+09 _runtime‚4658.952799207 _step‚437 & lr-AdamW/pg1‚0.0004951649674228867 & lr-AdamW/pg2‚0.0004951649674228867J¦ÌÌÆ & lr-AdamW/pg1‚0.0004951649674228867 & lr-AdamW/pg2‚0.0004951649674228867 trainer/global_step‚5349 % _timestamp‚1.7843673804459164e+09 _runtime‚4658.952799207 _step‚437µ%ù‚2Í- _wandbruntime‚4658  _runtime‚4658 4¨tâ'ÎÜO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49547357 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚179174694299648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20885879 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚814326894243098 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5646911167770 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚812859715969024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚818515240296448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5655541366784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11511749 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42403718 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5655541366784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20886140 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9374388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941098 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42403720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9374391 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11511491 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4658.953236092 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882705920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179174675687424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537380 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚5349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49547357 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5655882705920 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240296448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91951077 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184889792783872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5715009907200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚808679983075328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5647251717660 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42403720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184889685594624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843673804468093e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5655541366784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11511749 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42403718 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5655541366784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20886140 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9374388 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91941098 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42403720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚812859698929664 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9374391 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11511491 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚812859715969024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 _runtime‚4658.953236092 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5655882705920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚179174675687424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚438 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49537380 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808680000114688 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 trainer/global_step‚5349 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49547357 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5655882705920 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818515240296448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91951077 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184889792783872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5715009907200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚808679983075328 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5647251717660 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42403720 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184889685594624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843673804468093e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5646911167770 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚812859715969024¶>jv2Ð- _wandbruntime‚4659  _runtime‚4659Àèˆë&ÑåM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 $ _timestamp‚1.784367381080951e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5647922221492 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚814474415968692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49546432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553993728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5656553993728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818663852338688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚808826510786560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852338688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895340032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5716117550080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11513471 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179208104445952 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚813006939959296 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚813006956998656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11513739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42411350 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49556423 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9376133 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20889875 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5648262771494 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42411352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20889604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179208123058176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚4659.587088898 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20889604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚179208123058176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 _runtime‚4659.587088898 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5647922221492 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚476 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚814474415968692 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49546432 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5656553993728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5656553993728 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818663852338688 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚808826510786560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808826493747200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚818663852338688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5656895340032 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9376136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5716117550080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11513471 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚179208104445952 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚813006939959296 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚813006956998656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11513739 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002·Jqó<2Ó- _wandbruntime‚4659  _runtime‚4659‹v«Ô‚ 3loss/FBt2/assignment_loss‚0.0054814995848333735 .loss/FRt2/detection_loss‚1.586085804375382 epoch‚13 % _timestamp‚1.7843673810826697e+09 _runtime‚4659.588343496 _step‚440 2loss/SRqqt2/assignment_loss‚0.5149867770706323 /loss/FRt1/detection_loss‚0.5627294353868194 0loss/SRqqt2/detection_loss‚0.700688526562927 /loss/FBt1/detection_loss‚0.7069279600341781 /loss/FBt2/detection_loss‚0.7353858133364055 2loss/SRqqt1/assignment_loss‚0.6298655128497053 2loss/FBt1/assignment_loss‚0.004323465443320629 1loss/SRqqt1/detection_loss‚1.0953574223921267 trainer/global_step‚5349 /loss/FRt1/assignment_loss‚1.923839063992254 0loss/FRt2/assignment_loss‚0.9122389657341916 %loss/total_loss‚9.377910246762776¦²# Õ‡ trainer/global_step‚5349 /loss/FRt1/assignment_loss‚1.923839063992254 0loss/FRt2/assignment_loss‚0.9122389657341916 %loss/total_loss‚9.377910246762776 3loss/FBt2/assignment_loss‚0.0054814995848333735 .loss/FRt2/detection_loss‚1.586085804375382 epoch‚13 % _timestamp‚1.7843673810826697e+09 _runtime‚4659.588343496 _step‚440 2loss/SRqqt2/assignment_loss‚0.5149867770706323 /loss/FRt1/detection_loss‚0.5627294353868194 0loss/SRqqt2/detection_loss‚0.700688526562927 /loss/FBt1/detection_loss‚0.7069279600341781 /loss/FBt2/detection_loss‚0.7353858133364055 2loss/SRqqt1/assignment_loss‚0.6298655128497053 2loss/FBt1/assignment_loss‚0.004323465443320629 1loss/SRqqt1/detection_loss‚1.0953574223921267¸çqÖ¯“Ö‚(:ˆ š’íÒݯ€í gpu.0.gpu‚42.0 gpu.0.memory‚15+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÕFºÅׂ(:º š’íÒ¢³ùï network.recv‚ 1.253118e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 6306.921875 disk.sda2.out‚0( disk./.usagePercent‚39.5885865226001 disk.rbd1.out‚12.0625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.out‚ 101.88671875$ disk./.usageGB‚589.8985481262207" disk.dm-0.out‚9267.47314453125 disk.rbd1.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 484.1484375 network.sent‚ 8.581953e+06% disk.nvme0n1.out‚9267.47314453125' disk.nvme0c0n1.out‚9267.47314453125 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 6306.921875 disk.sda3.out‚ 101.88671875 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚97.04499244689941" proc.memory.rssMB‚ 2797.0078125 cpu‚26.020979323461578 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 6306.921875 disk.sda.out‚ 101.88671875) proc.memory.percent‚17.07158088684082g©·’Ø‚(:‡ ©’íÒ”·´î gpu.0.gpu‚24.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊBo»Ù‚(:° ©’íÒúŠýï$ disk./.usageGB‚589.8986015319824 disk.dm-0.out‚ 9271.47265625# disk.nvme0c0n1.in‚ 6391.25390625 disk.sda.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 102.2109375 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2797.0078125) proc.memory.percent‚17.07158088684082) disk./.usagePercent‚39.58859010670558 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚26.02197488955748# proc.memory.availableMB‚483.875 network.sent‚ 8.644843e+06 disk.drbd2165.in‚0 disk.dm-0.in‚ 6391.25390625 disk.sda2.out‚0! disk.nvme0n1.in‚ 6391.25390625 disk.sda2.in‚0 disk.sda.out‚ 102.2109375 disk.sda3.out‚ 102.2109375 disk.rbd1.out‚12.0625 network.recv‚ 1.257908e+06$ memory_percent‚97.04666137695312 disk.sda3.in‚0" disk.nvme0n1.out‚ 9271.47265625$ disk.nvme0c0n1.out‚ 9271.47265625 disk.sda1.in‚0KA{‘Ú‚(:† ¸’íÒ¯µÊî gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚277.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ™ »ÇÛ‚(:¼ ¸’íÒŒ˜Éï disk.sda.out‚ 102.4921875 disk.sda1.in‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚9276.26123046875$ memory_percent‚97.04678058624268 disk.dm-0.in‚ 6391.25390625" disk.dm-0.out‚9276.26123046875) disk./.usagePercent‚39.58859778693161' disk.nvme0c0n1.out‚9276.26123046875 disk.sda.in‚0 disk.sda3.out‚ 102.4921875 network.recv‚ 1.261125e+06$ disk./.usageGB‚589.8987159729004# disk.nvme0c0n1.in‚ 6391.25390625" proc.memory.rssMB‚ 2797.0078125 network.sent‚ 8.649711e+06 disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 6391.25390625( proc.memory.availableMB‚ 483.85546875) proc.memory.percent‚17.07158088684082 cpu‚26.0231279883998 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚12.125 disk.dm-1.out‚ 102.4921875 disk.drbd2165.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0hõ à2Ü- _wandbruntime‚4700  _runtime‚4700,}ÇÝÁ & lr-AdamW/pg2‚0.0004934963830293547 trainer/global_step‚5399 % _timestamp‚1.7843674220807948e+09 _runtime‚4700.587806545 _step‚441 & lr-AdamW/pg1‚0.0004934963830293547B·ñçÌÞÆ & lr-AdamW/pg1‚0.0004934963830293547 & lr-AdamW/pg2‚0.0004934963830293547 trainer/global_step‚5399 % _timestamp‚1.7843674220807948e+09 _runtime‚4700.587806545 _step‚441¹‚9ª2ß- _wandbruntime‚4700  _runtime‚47004¹VÉâ'àÜO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42785320 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 _runtime‚4700.587908247 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚820221750411264 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚180846132226048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚816005516668928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5706236868608 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5765972100608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚825927987279872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9461638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5399 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5706578207744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21076463 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180846113613824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚820221767450624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5697868204568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987279872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49991832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186612085714432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5766062774784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821703401912856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49991832 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42785318 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9461641 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚820221750411264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42785318 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186612195000832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825928345658368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11614822 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820221767450624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323606 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825928345658368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42785320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236868608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5697527654678 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11614559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚820221767450624 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821703044323606 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚825928345658368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 109286400 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42785320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5706236868608 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5697527654678 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11614559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5765972100608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚825927987279872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9461638 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚5399 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5706578207744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21076463 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180846113613824 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚820221767450624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5697868204568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚825927987279872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49991832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚186612085714432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5766062774784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚821703401912856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚442 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceSt.¶Ó{§ atsMonitor.on_train_batch_start/active.small_pool.freed‚49991832 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42785318 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9461641 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚541 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚820221750411264 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42785318 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚186612195000832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚825928345658368 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11614822ºˆ¤´2â- _wandbruntime‚4701  _runtime‚4701ðï€8í&ãçM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5707590841856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599322112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820369008480256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42792952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚820368991440896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50010875 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50000884 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186646645061632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5707249495552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42792950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11616887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11616604 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚5399 _runtime‚4701.216958033 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186646521102336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9463386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5707249495552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000884 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21079987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5767084077056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5766978729984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42792952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21080273¬I«‚ò&äìM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42792952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21080273 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5707590841856 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826076599322112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚820369008480256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚816152044380160 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42792952 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚541 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 105347072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821850923638562 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚820368991440896 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚443 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 123959296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50010875 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50000884 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚186646645061632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5707249495552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42792950 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11616887 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11616604 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 trainer/global_step‚5399 _runtime‚4701.216958033 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚186646521102336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9463386 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821850566049200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5707249495552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50000884 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21079987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5767084077056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5766978729984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92793834 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816152027340800»sÞή2å- _wandbruntime‚4701  _runtime‚4701”£D æ„ 0loss/FRt2/assignment_loss‚0.9007297216085443 2loss/FBt1/assignment_loss‚0.004792199276793263 /loss/FBt2/detection_loss‚0.6913736747916449 0loss/FRt1/assignment_loss‚1.9653080278786563 2loss/SRqqt1/assignment_loss‚0.5947009474580545 1loss/SRqqt1/detection_loss‚1.1266134734503752 epoch‚13 _step‚444 2loss/SRqqt2/assignment_loss‚0.5553083681672374 3loss/FBt2/assignment_loss‚0.0019975295199123684 .loss/FBt1/detection_loss‚0.577036933552119 % _timestamp‚1.7843674227124522e+09 /loss/FRt1/detection_loss‚0.5215865620438754 /loss/FRt2/detection_loss‚1.3901505859712147 1loss/SRqqt2/detection_loss‚0.6184903759204379 %loss/total_loss‚8.948088399638866 trainer/global_step‚5399 _runtime‚4701.218127232dxç‰ 0loss/FRt1/assignment_loss‚1.9653080278786563 2loss/SRqqt1/assignment_loss‚0.5947009474580545 1loss/SRqqt1/detection_loss‚1.1266134734503752 epoch‚13 _step‚444 2loss/SRqqt2/assignment_loss‚0.5553083681672374 3loss/FBt2/assignment_loss‚0.0019975295199123684 .loss/FBt1/detection_loss‚0.577036933552119 % _timestamp‚1.7843674227124522e+09 /loss/FRt1/detection_loss‚0.5215865620438754 /loss/FRt2/detection_loss‚1.3901505859712147 1loss/SRqqt2/detection_loss‚0.6184903759204379 %loss/total_loss‚8.948088399638866 trainer/global_step‚5399 _runtime‚4701.218127232 0loss/FRt2/assignment_loss‚0.9007297216085443 2loss/FBt1/assignment_loss‚0.004792199276793263 /loss/FBt2/detection_loss‚0.6913736747916449¼™jGØžè‚(:“ Ç’íÒ‘æÒí gpu.0.gpu‚77.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚259.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.821799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù}%Ëé‚(:À Ç’íÒÏ‘—ð) disk./.usagePercent‚39.58859727491654 disk.rbd1.in‚0 disk.rbd1.out‚12.125 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚589.8987083435059 disk.dm-0.in‚ 6391.25390625' proc.memory.availableMB‚ 483.8046875 disk.dm-1.in‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚9279.37158203125 disk.sda2.in‚0" proc.memory.rssMB‚ 2797.0078125 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚9279.37158203125 disk.sda.out‚ 102.79296875 network.sent‚ 8.654635e+06 disk.dm-1.out‚ 102.79296875# disk.nvme0c0n1.in‚ 6391.25390625$ memory_percent‚97.04709053039551 network.recv‚ 1.264414e+06! disk.nvme0n1.in‚ 6391.25390625" disk.dm-0.out‚9279.37158203125) proc.memory.percent‚17.07158088684082 disk.sda3.in‚0 disk.sda3.out‚ 102.79296875 disk.sda.in‚0 cpu‚26.024156710105718FÛ_Cžê‚(:“ Ö’íÒµ‘þî gpu.0.gpu‚89.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚232.724' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚46.544799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜(Æë‚(:» Ö’íÒÄíðð disk.drbd2165.out‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 6391.25390625! disk.dm-0.out‚9285.0791015625' proc.memory.availableMB‚ 483.3359375 proc.cpu.threads‚17 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2797.0078125$ disk.nvme0n1.out‚9285.0791015625 disk.sda1.out‚0 cpu‚26.025177880977132) disk./.usagePercent‚39.58859317879599 disk.dm-0.in‚ 6391.25390625 disk.sda3.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.07158088684082 disk.sda3.out‚ 103.1640625 disk.dm-1.out‚ 103.1640625 network.sent‚ 8.719653e+06 disk.sda2.in‚0 disk.rbd1.out‚12.1875& disk.nvme0c0n1.out‚9285.0791015625 disk.sda1.in‚0$ memory_percent‚97.04995155334473 network.recv‚ 1.268773e+06$ disk./.usageGB‚589.8986473083496 disk.rbd0.in‚0 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6391.25390625 disk.sda.out‚ 103.1640625 disk.sda2.out‚0 disk.sda.in‚0bµ Ñ“ì‚(:ˆ å’íÒáû¬í gpu.0.gpu‚38.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚197.728' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.5456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥˜ Êí‚(:¿ å’íÒ‚ƒ¹ï# disk.nvme0c0n1.in‚ 6509.91796875 disk.sda1.in‚0 disk.dm-1.out‚ 103.41015625 cpu‚26.026321583714537 disk.rbd0.in‚0 disk.drbd2165.out‚0 network.recv‚ 1.272428e+06 disk.sda2.out‚0 disk.sda.out‚ 103.41015625 network.sent‚ 8.722718e+06$ disk./.usageGB‚589.8986549377441 disk.sda3.out‚ 103.41015625 disk.dm-1.in‚0& proc.memory.availableMB‚ 483.578125 proc.cpu.threads‚17) proc.memory.percent‚17.07158088684082 disk.rbd0.out‚0" disk.dm-0.out‚9287.41943359375 disk.rbd1.out‚12.1875' disk.nvme0c0n1.out‚9287.41943359375 disk.sda3.in‚0# memory_percent‚97.0484733581543! disk.nvme0n1.in‚ 6509.91796875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2797.0078125) disk./.usagePercent‚39.58859369081106% disk.nvme0n1.out‚9287.41943359375 disk.dm-0.in‚ 6509.91796875 disk.sda1.out‚0Ö…2î- _wandbruntime‚4742  _runtime‚4742•Å6]ÇïÁ trainer/global_step‚5449 % _timestamp‚1.7843674635728314e+09 _runtime‚4742.078867396 _step‚445 & lr-AdamW/pg1‚0.0004918147934009082 & lr-AdamW/pg2‚0.00049181479340090828ÖøÌðÆ _runtime‚4742.078867396 _step‚445 & lr-AdamW/pg1‚0.0004918147934009082 & lr-AdamW/pg2‚0.0004918147934009082 trainer/global_step‚5449 % _timestamp‚1.7843674635728314e+09½Ni·&2ñ- _wandbruntime‚4742  _runtime‚4742é¬"^ òÝO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50446284 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734263296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141586 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5817025789440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50456261 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273709568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833341092641792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11717502 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5748484691476 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93623181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11717249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613202 _runtime‚4742.082723505 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43166920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43166918 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827583801892864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21266137 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚5449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚833341092641792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188334595941888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833340734263296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932370432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169þšiè'óâO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43166920 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚823331050262528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50446284 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚833340734263296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5748144141586 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5817025789440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50456261 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5757273709568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚833341092641792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11717502 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 101159936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5748484691476 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93623181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11717249 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93613202 _runtime‚4742.082723505 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43166920 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43166918 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827583801892864 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21266137 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182517570152448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 trainer/global_step‚5449 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚833341092641792 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9548888 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚188334595941888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚833340734263296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚446 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5756932370432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169¾Òðù2ô- _wandbruntime‚4742  _runtime‚4742á˜rì&õæM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5758286343680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833489346305536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827731042922496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11719252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833489346305536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5749155195308 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827731042922496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188368929387520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43174550 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11719528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5817949088768 % _timestamp‚1.7843674642063494e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21270164 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21269885 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43174552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182550980298752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚829227073719070 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188369040763904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4742.712445816 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 5539679837×™½RöëM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5818041852928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50455336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5757944997376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 _step‚447 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639879 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827731042922496 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9550633 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚276 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚188368929387520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43174550 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11719528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5817949088768 % _timestamp‚1.7843674642063494e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21270164 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21269885 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43174552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182550980298752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚829227073719070 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚188369040763904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4742.712445816 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5758286343680 C=DeviceS‹ÐzßtatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚833489346305536 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182550998910976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827731042922496 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11719252 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚833489346305536 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5749155195308 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840753839625216 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43548518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184189008078848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5799101178384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834945870413824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43548518 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807969211392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840753481246720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184188989466624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834945870413824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50900736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9636138 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11819804 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843675052348928e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834945853374464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21455942 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5867979869184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807969211392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481246720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚830656583856128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50900736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628494 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910713 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9636141 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169,ácç'ƒáO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 113480704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚836455344484622 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5807627872256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _runtime‚4783.741035466 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚190056874467328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚830656600895488 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5807969211392 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚840753481246720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚830656583856128 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50900736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5798760628494 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50910713 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9636141 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43548520 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚836455702073872 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50910713 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21456214 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚840753839625216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459233 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚840753839625216 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43548518 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184189008078848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5799101178384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚834945870413824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43548518 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5807969211392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚840753481246720 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184188989466624 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚834945870413824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚94868480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50900736 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9636138 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11819804 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 % _timestamp‚1.7843675052348928e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚834945853374464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21455942 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5867979869184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835093111443456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚835093111443456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11821803 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 _runtime‚4784.372556108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11822076 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5799771682216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5808640499200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21459962 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 $ _timestamp‚1.784367505866403e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5808640499200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190091315200512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5868998128128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43556152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50909788 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840902093288960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚451 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50919779 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184222418225152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50919779 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835093094404096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210216 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190091434965504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21459686 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144äz'.ñ&†ëM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835093094404096 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚836602866210216 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚190091434965504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21459686 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43556150 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚830803094528000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835093111443456 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚835093111443456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11821803 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 101152768 _runtime‚4784.372556108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11822076 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5799771682216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5808640499200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21459962 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 $ _timestamp‚1.784367505866403e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5808640499200 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚190091315200512 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5868998128128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43556152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50909788 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚840902093288960 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 _step‚451 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50919779 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184222418225152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50919779 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 119764992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0Ãbb³ˆ2‡- _wandbruntime‚4784  _runtime‚4784{‹(Ê ˆ„ 1loss/SRqqt2/detection_loss‚0.6720906338211425 epoch‚13 trainer/global_step‚5499 _runtime‚4784.373872423 2loss/SRqqt2/assignment_loss‚0.5602659022721252 1loss/FBt1/assignment_loss‚0.01510369842595971 0loss/FBt2/detection_loss‚0.38878990232103294 %loss/total_loss‚8.205414784771554 % _timestamp‚1.7843675058681788e+09 _step‚452 0loss/FRt1/assignment_loss‚1.8844353746568132 /loss/FRt2/detection_loss‚1.3232024845763604 2loss/FBt2/assignment_loss‚0.008777214996371185 0loss/SRqqt1/detection_loss‚0.883081535769937 0loss/FBt1/detection_loss‚0.39621633474449935 0loss/FRt2/assignment_loss‚0.9295302243860624 2loss/SRqqt1/assignment_loss‚0.5889305024951076 /loss/FRt1/detection_loss‚0.5549909763061428$fé ‰‰ /loss/FRt1/detection_loss‚0.5549909763061428 1loss/SRqqt2/detection_loss‚0.6720906338211425 epoch‚13 trainer/global_step‚5499 _runtime‚4784.373872423 2loss/SRqqt2/assignment_loss‚0.5602659022721252 1loss/FBt1/assignment_loss‚0.01510369842595971 0loss/FBt2/detection_loss‚0.38878990232103294 %loss/total_loss‚8.205414784771554 % _timestamp‚1.7843675058681788e+09 _step‚452 0loss/FRt1/assignment_loss‚1.8844353746568132 /loss/FRt2/detection_loss‚1.3232024845763604 2loss/FBt2/assignment_loss‚0.008777214996371185 0loss/SRqqt1/detection_loss‚0.883081535769937 0loss/FBt1/detection_loss‚0.39621633474449935 0loss/FRt2/assignment_loss‚0.9295302243860624 2loss/SRqqt1/assignment_loss‚0.5889305024951076ÄåÊêã‘Š‚(:† ’“íÒÉ áî gpu.0.gpu‚88.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚231.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I`Ë‹‚(:À ’“íÒˆ˜õð' disk.nvme0c0n1.out‚9299.13134765625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚0# memory_percent‚97.0508337020874) proc.memory.percent‚17.07158088684082 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 cpu‚26.029229386243355 proc.cpu.threads‚17 disk.sda3.out‚ 104.33984375 disk.dm-1.in‚0 disk.dm-1.out‚ 104.33984375# disk.nvme0c0n1.in‚ 6509.91796875 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚12.375 network.recv‚ 1.283336e+06 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 104.33984375( proc.memory.availableMB‚ 483.19140625" proc.memory.rssMB‚ 2797.0078125$ disk./.usageGB‚590.6964874267578" disk.dm-0.out‚9299.13134765625 network.sent‚ 8.795432e+06) disk./.usagePercent‚39.64213689857481 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 6509.91796875% disk.nvme0n1.out‚9299.13134765625 disk.dm-0.in‚ 6509.91796875†Ùc°Œ‚(:’ ¡“íÒÑâ”ï gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚260.765' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.15299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÍM2‚(:· ¡“íÒƸ‹ñ proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 network.sent‚ 8.860328e+06& disk.nvme0n1.out‚10116.03173828125 disk.dm-0.in‚ 6608.3515625# disk.dm-0.out‚10116.03173828125 disk.rbd1.in‚0 disk.sda.out‚ 104.65625' proc.memory.availableMB‚ 482.8984375 disk.dm-1.out‚ 104.65625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6608.3515625 disk.sda2.out‚0( disk.nvme0c0n1.out‚10116.03173828125) proc.memory.percent‚17.07158088684082 cpu‚26.030276776710572 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚39.64015463223719 disk.sda3.out‚ 104.65625" disk.nvme0c0n1.in‚ 6608.3515625 disk.sda.in‚0" proc.memory.rssMB‚ 2797.0078125 network.recv‚ 1.288026e+06$ disk./.usageGB‚590.6669502258301 disk.sda3.in‚0 disk.rbd1.out‚12.375$ memory_percent‚97.05262184143066Ý«ì“Ž‚(:ˆ °“íÒäÃöí gpu.0.gpu‚72.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚228.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.7372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e*“ƒÆ‚(:» °“íÒïÞ¿ð network.recv‚ 1.291247e+06! disk.dm-0.out‚10120.689453125 disk.nvme0n1.in‚ 6608.3515625$ disk.nvme0n1.out‚10120.689453125 disk.dm-0.in‚ 6608.3515625& disk.nvme0c0n1.out‚10120.689453125 disk.sda3.out‚ 104.87890625 disk.rbd0.in‚0 network.sent‚ 8.865313e+06) disk./.usagePercent‚39.59223309391782 disk.drbd2165.in‚0$ memory_percent‚97.05262184143066" proc.memory.rssMB‚ 2797.0078125 disk.rbd1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚12.4375 disk.sda.in‚0 disk.sda.out‚ 104.87890625 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 104.87890625 disk.drbd2165.out‚0) proc.memory.percent‚17.07158088684082' proc.memory.availableMB‚ 482.8984375$ disk./.usageGB‚589.9528846740723 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 6608.3515625 disk.sda3.in‚0 cpu‚26.031428816985425 disk.sda2.in‚0 disk.sda1.in‚0ÈËô2- _wandbruntime‚4825  _runtime‚4825akZ¥Ç‘Á & lr-AdamW/pg2‚0.0004884130312890748 trainer/global_step‚5549 % _timestamp‚1.7843675468958256e+09 _runtime‚4825.401803532 _step‚453 & lr-AdamW/pg1‚0.0004884130312890748{µv»Ì’Æ % _timestamp‚1.7843675468958256e+09 _runtime‚4825.401803532 _step‚453 & lr-AdamW/pg1‚0.0004884130312890748 & lr-AdamW/pg2‚0.0004884130312890748 trainer/global_step‚5549ż¼€¾2“- _wandbruntime‚4825  _runtime‚4825 ×Åö`%”ÜO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323374080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842307904856064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5919068002304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5918985716736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43930120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837982134489088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295285 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843831494565130 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11922357 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365165 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848166586608640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43930118 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228230144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191779514007552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9723388 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185860446005248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438ÜÂç'•áO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚842307921895424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚842307904856064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5858664713216 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9723391 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51365165 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5849377115402 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚848166586608640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 trainer/global_step‚5549 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51355188 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚454 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51355188 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21646001 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5858323374080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚842307904856064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5919068002304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5918985716736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43930120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚837982134489088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295285 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843831494565130 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11922357 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51365165 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚848166586608640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43930118 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚848166228230144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚82285568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191779514007552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9723388 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185860446005248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5849717665292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5920099053056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚838128645160960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5920020968960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚843979373880086 trainer/global_step‚5549 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11924418 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚842455145885696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5859336001024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21649551 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11924670 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5859677347328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43937750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842455162925056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301990 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚848314840272384 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5850728719126 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43937750 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚848314840272384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5859677347328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191813973816832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51364240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185893856151552 _runtime‚4826.037793133 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843979016290724 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21649551 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11924670 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43937752 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5859677347328 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43937750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚842455162925056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚95301990 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚848314840272384 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5850728719126 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43937750 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚848314840272384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5859677347328 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51374231 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191813973816832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51364240 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185893856151552 _runtime‚4826.037793133 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚843979016290724 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43937752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5920099053056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚838128645160960 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 SFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚96696320 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5920020968960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438ǵ•ì2™- _wandbruntime‚4826  _runtime‚4826ÚYš‚ 0loss/FRt1/assignment_loss‚1.8957405858803986 2loss/SRqqt1/assignment_loss‚0.5348544664475627 3loss/FBt1/assignment_loss‚0.0006678806275201416 /loss/FRt1/detection_loss‚0.5991181849906702 /loss/FRt2/detection_loss‚1.4919076290053828 _runtime‚4826.039069735 _step‚456 1loss/SRqqt1/detection_loss‚0.8768068521657563 /loss/FBt2/detection_loss‚1.1141654670936987 %loss/total_loss‚9.081272824234372 epoch‚13 trainer/global_step‚5549 1loss/FBt2/assignment_loss‚0.02598585120105379 1loss/SRqqt2/detection_loss‚0.6152468136056746 $ _timestamp‚1.784367547533383e+09 0loss/FRt2/assignment_loss‚0.8290655580621996 1loss/SRqqt2/assignment_loss‚0.554490910818956 /loss/FBt1/detection_loss‚0.5432226243354972MÞ–’ ›‡ 3loss/FBt1/assignment_loss‚0.0006678806275201416 /loss/FRt1/detection_loss‚0.5991181849906702 /loss/FRt2/detection_loss‚1.4919076290053828 _runtime‚4826.039069735 _step‚456 1loss/SRqqt1/detection_loss‚0.8768068521657563 /loss/FBt2/detection_loss‚1.1141654670936987 %loss/total_loss‚9.081272824234372 epoch‚13 trainer/global_step‚5549 1loss/FBt2/assignment_loss‚0.02598585120105379 1loss/SRqqt2/detection_loss‚0.6152468136056746 $ _timestamp‚1.784367547533383e+09 0loss/FRt2/assignment_loss‚0.8290655580621996 1loss/SRqqt2/assignment_loss‚0.554490910818956 /loss/FBt1/detection_loss‚0.5432226243354972 0loss/FRt1/assignment_loss‚1.8957405858803986 2loss/SRqqt1/assignment_loss‚0.5348544664475627Èî;¯œ‚(:ˆ ¿“íÒ›­€î gpu.0.gpu‚97.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚265.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powWÁ)erPercent‚53.1686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•³hQÂ(:¸ ¿“íÒáÛÝï disk.nvme0n1.in‚ 6608.3515625" disk.nvme0c0n1.in‚ 6608.3515625 disk.sda.out‚ 105.16796875" proc.memory.rssMB‚ 2797.0078125# disk.nvme0n1.out‚10124.17578125 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚97.05414772033691 disk.drbd2165.out‚0 disk.dm-0.out‚10124.17578125 disk.rbd1.in‚0 disk.rbd1.out‚12.4375 disk.sda.in‚0 cpu‚26.032370352400637 proc.cpu.threads‚17$ disk./.usageGB‚589.9521179199219 disk.drbd2165.in‚0) disk./.usagePercent‚39.59218163640343 disk.sda2.in‚0% disk.nvme0c0n1.out‚10124.17578125) proc.memory.percent‚17.07158088684082' proc.memory.availableMB‚ 482.6484375 disk.dm-0.in‚ 6608.3515625 network.sent‚ 8.868336e+06 network.recv‚ 1.294855e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 105.16796875 disk.sda1.in‚0 disk.sda3.out‚ 105.16796875ùÚW“ž‚(:ˆ ΓíÒë¨Âí gpu.0.gpu‚87.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03q.(¿Ÿ‚(:´ ΓíÒÝ­Æï disk.sda.out‚ 105.5703125 disk.dm-1.in‚0$ memory_percent‚97.05379009246826 disk.dm-0.out‚10131.89453125 network.sent‚ 8.931264e+06 network.recv‚ 1.299156e+06 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 105.5703125 disk.rbd0.out‚0 disk.dm-0.in‚ 6608.3515625 disk.rbd1.out‚12.5# disk.nvme0n1.out‚10131.89453125" proc.memory.rssMB‚ 2797.0078125 cpu‚26.033401965319815 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 6608.3515625% disk.nvme0c0n1.out‚10131.89453125 disk.sda.in‚0 disk.sda3.in‚0* disk./.usagePercent‚39.590443345245944 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 105.5703125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 6608.3515625) proc.memory.percent‚17.07158088684082( proc.memory.availableMB‚ 482.70703125$ disk./.usageGB‚589.9262161254883 disk.drbd2165.in‚0H€8c ‚(:’ Ý“íÒÅÙåî gpu.0.gpu‚86.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚39.0 gpu.0.powerWatts‚241.325' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚48.26499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ¯•jË¡‚(:À Ý“íÒ¦âˆñ# disk.nvme0c0n1.in‚ 6661.70703125 disk.dm-1.out‚ 105.7890625 disk.rbd0.out‚0 network.recv‚ 1.302795e+06( disk.nvme0c0n1.out‚10136.28955078125) proc.memory.percent‚17.07160472869873 proc.cpu.threads‚17 network.sent‚ 8.938091e+06$ disk./.usageGB‚589.9262886047363 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚ 6661.70703125' proc.memory.availableMB‚ 482.1640625 cpu‚26.034203834963368& disk.nvme0n1.out‚10136.28955078125# disk.dm-0.out‚10136.28955078125* disk./.usagePercent‚39.590448209389095 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚97.05710411071777 disk.rbd1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 105.7890625! disk.nvme0n1.in‚ 6661.70703125# proc.memory.rssMB‚ 2797.01171875 disk.rbd1.out‚12.5 disk.sda3.out‚ 105.7890625 disk.dm-1.in‚0 disk.rbd0.in‚0¦,\2¢- _wandbruntime‚4866  _runtime‚4866¥P‡çÇ£Á _step‚457 & lr-AdamW/pg1‚0.0004866930772945115 & lr-AdamW/pg2‚0.0004866930772945115 trainer/global_step‚5599 % _timestamp‚1.7843675884048262e+09 _runtime‚4866.910750042‡ªþÌ¤Æ % _timestamp‚1.7843675884048262e+09 _runtime‚4866.910750042 _step‚457 & lr-AdamW/pg1‚0.0004866930772945115 & lr-AdamW/pg2‚0.0004866930772945115 trainer/global_step‚5599É]í62¥- _wandbruntime‚4866  _runtime‚4866-Ïmâ'¦ÜO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5909360215040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843675884058921e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5970063412224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚849669973377024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12025675 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5899993602310 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚849669956337664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5909018875904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21836583 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193501928731648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51809640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851208002234888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333592064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5970151989248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚5599 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44311720 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187531865319424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5909018875904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9810641 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193502035920896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845307668082688 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚4866.912068792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5909360215040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187531883931648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚855579333592064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845307651043328 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚849669973377024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855578975213568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21836313 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5900334152200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12025942 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51819617 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975213568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5909360215040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 % _timestamp‚1.7843675884058921e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5970063412224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚849669973377024 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44311718 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12025675 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5899993602310 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚849669956337664 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5909018875904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44311718 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21836583 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193501928731648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51809640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚851208002234888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚855579333592064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5970151989248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚5599 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44311720 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187531865319424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5909018875904 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9810641 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚193502035920896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚845307668082688 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 _runtime‚4866.912068792 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5909360215040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187531883931648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚855579333592064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 107189248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚845307651043328 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚849669973377024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚855578975213568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44311720 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51809640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚88577024 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21836313 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5900334152200 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12025942 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96131337 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51819617 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚855578975213568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚5599 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193536379471360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚849817197367296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51818692 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51828683 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚845454178754560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855727587255808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587255808 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5910372849152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5901345206034 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44319350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44319352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372849152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4867.547769217 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5971173963264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5971085393408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784367589041547e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51828683 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚849817197367296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44319352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5901004656032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21840047 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5910031502848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187565312690176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12027664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚459 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚851355523960594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚849817214406656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088Pc>ð&ªêM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚845454161715200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚5599 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193536379471360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚849817197367296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51818692 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51828683 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚845454178754560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96138042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚855727587255808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚855727587255808 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5910372849152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5901345206034 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44319350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44319352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5910372849152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4867.547769217 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9812386 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5971173963264 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5971085393408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 $ _timestamp‚1.784367589041547e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51828683 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚849817197367296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44319352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5901004656032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21840047 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5910031502848 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44319350 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187565312690176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12027664 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚459 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚851355523960594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚849817214406656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12027927 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51818692 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5910031502848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21840313 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9812383 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983Ë{eƒ?2«- _wandbruntime‚4867  _runtime‚4867Ä6Õh ¬… 2loss/FBt1/assignment_loss‚0.005527437048616526 /loss/FRt1/detection_loss‚0.5891112850330611 /loss/FBt2/detection_loss‚0.6772067423111626 %loss/total_loss‚8.788635119538235 % _timestamp‚1.7843675890428195e+09 0loss/FRt1/assignment_loss‚1.8957163864224527 /loss/FRt2/assignment_loss‚0.884785071437658 2loss/SRqqt2/assignment_loss‚0.5134111806976838 /loss/FBt1/detection_loss‚0.4921351998614577 _runtime‚4867.548636064 4loss/FBt2/assignment_loss‚3.8421098423415045e-05 /loss/FRt2/detection_loss‚1.3374759358731438 1loss/SRqqt2/detection_loss‚0.6548276794377353 epoch‚13 trainer/global_step‚5599 _step‚460 2loss/SRqqt1/assignment_loss‚0.7078166063214486 1loss/SRqqt1/detection_loss‚1.0305831739953908‹±Œy­Š 2loss/FBt1/assignment_loss‚0.005527437048616526 /loss/FRt1/detection_loss‚0.5891112850330611 /loss/FBt2/detection_loss‚0.6772067423111626 %loss/total_loss‚8.788635119538235 % _timestamp‚1.7843675890428195e+09 0loss/FRt1/assignment_loss‚1.8957163864224527 /loss/FRt2/assignment_loss‚0.884785071437658 2loss/SRqqt2/assignment_loss‚0.5134111806976838 /loss/FBt1/detection_loss‚0.4921351998614577 _runtime‚4867.548636064 4loss/FBt2/assignment_loss‚3.8421098423415045e-05 /loss/FRt2/detection_loss‚1.3374759358731438 1loss/SRqqt2/detection_loss‚0.6548276794377353 epoch‚13 trainer/global_step‚5599 _step‚460 2loss/SRqqt1/assignment_loss‚0.7078166063214486 1loss/SRqqt1/detection_loss‚1.0305831739953908Ì)8˜Ð“®‚(:ˆ ì“í҈ŵí gpu.0.gpu‚79.0 gpu.0.memory‚21+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚278.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñÚį‚(:¹ ì“íÒ°Ó¤ï disk.dm-0.in‚ 6710.12109375 disk.sda1.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 8.941013e+06) disk./.usagePercent‚39.59044155319321 disk.rbd1.in‚0 disk.rbd1.out‚12.5625& proc.memory.availableMB‚ 476.578125! disk.nvme0n1.in‚ 6710.12109375# disk.dm-0.out‚10142.53369140625 disk.dm-1.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2802.28125$ disk./.usageGB‚589.9261894226074# disk.nvme0c0n1.in‚ 6710.12109375 disk.dm-1.out‚ 106.015625 disk.drbd2165.in‚0# memory_percent‚97.0911979675293( disk.nvme0c0n1.out‚10142.53369140625 disk.sda3.out‚ 106.015625) proc.memory.percent‚17.10376739501953 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚ 1.306201e+06 disk.sda.out‚ 106.015625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda.in‚0 cpu‚26.03998639457223& disk.nvme0n1.out‚10142.53369140625ò•Î%2°- _wandbruntime‚4885  _runtime‚4885#Ö'…_=±Ùz ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6815762519836426 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04489597678184509 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.684576153755188 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 *jet/accuracy_2_of_4‚0.7805314660072327 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963895678520203 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6475377082824707 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09112142026424408 (particle/accuracy‚0.8472880125045776 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7508991360664368 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10774707794189453 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24207952618598938 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6383215188980103 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.726094663143158 )jet/accuracy_1_of_2‚0.806817352771759 /particle/accuracy_3_of_4‚0.9904478788375854 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8402872681617737 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2668633759021759 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2956760823726654 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6822385191917419 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3405454754829407 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6350078582763672 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38441699743270874 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1847202032804489 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6432257294654846 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7316102385520935 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7946666479110718 +jet/accuracy_2_of_2‚0.15115192532539368 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942052960395813 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7036387920379639 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208134651184 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2851649820804596 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28430238366127014 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29592078924179077 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2856338918209076 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32607391476631165 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947041273117065 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41692793369293213 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.037391193211078644 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166865348816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6701688766479492 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35064536333084106 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40524160861968994 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18131308257579803 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960395097732544 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7606930732727051 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6345593333244324 $ _timestamp‚1.784367607235531e+09 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6462283134460449 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957683086395264 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8558579683303833 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9101465940475464 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5545833706855774 )loss/val_total_loss‚8.620866685439873 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996483564376831 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09112142026424408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1094556674361229 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2570221424102783 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6502059102058411 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18876683712005615 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987663745880127 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6082642078399658 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7312976121902466 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 jet/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46985912322998047 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.760745644569397 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5840864181518555 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6408697962760925 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49589696526527405 particle/accuracy_2_of_5‚1 _runtime‚4885.742167128 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8368104100227356 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24828195571899414 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7946666479110718 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27331215143203735 *jet/accuracy_3_of_3‚0.0767824798822403 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29643386602401733 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17224976420402527 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7377908825874329 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6420308351516724 +jet/accuracy_1_of_1‚0.35267025232315063 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6764509081840515 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4620387554168701 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7078264951705933 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8998740911483765 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6824098825454712 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10390318930149078 +jet/accuracy_3_of_4‚0.08877447992563248 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7355902194976807 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24759581685066223 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8497544527053833 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_4_of_5‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6235604286193848 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22606870532035828 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7679516077041626 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6424446105957031 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6259852647781372 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2856338918209076 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1580902338027954 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6475377082824707 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31586146354675293 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2764119505882263 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08509891480207443 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7524863481521606 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1030573844909668 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.645194411277771 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9492474794387817 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25891080498695374 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9391651153564453 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8701770305633545 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.72503262758255 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_4‚0.9671691656112671 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.5633708238601685 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5584731101989746 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3877018392086029 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6327316761016846 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7262282371520996 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0702979564666748 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚13 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17431291937828064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8601285815238953 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9247224926948547 /particle/accuracy_2_of_2‚0.9998962879180908 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6839922070503235 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6865627765655518 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05389911308884621 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041851580142974854 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970179796218872 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6174413561820984 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2591405510902405 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7validation_average_jet_accuracy‚0.39066410064697266 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874575734138489 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2872249186038971 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21634553372859955 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5613088011741638 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46969544887542725 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 trainer/global_step‚5599 _step‚461 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8779480457305908 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 particle/accuracy_2_of_3‚19 CÅJ²Þz @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7508991360664368 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10774707794189453 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.24207952618598938 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6383215188980103 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.726094663143158 )jet/accuracy_1_of_2‚0.806817352771759 /particle/accuracy_3_of_4‚0.9904478788375854 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8402872681617737 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2668633759021759 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2956760823726654 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6822385191917419 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3405454754829407 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6350078582763672 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38441699743270874 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1847202032804489 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6432257294654846 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9993972778320312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7316102385520935 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqtÕX½6*FBt/event_proportion‚0.0127883180975914 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7777777910232544 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7946666479110718 +jet/accuracy_2_of_2‚0.15115192532539368 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995930194854736 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942052960395813 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7036387920379639 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208134651184 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2851649820804596 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.28430238366127014 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.29592078924179077 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2856338918209076 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.32607391476631165 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947041273117065 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41692793369293213 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.037391193211078644 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166865348816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6701688766479492 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.35064536333084106 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40524160861968994 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18131308257579803 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9960395097732544 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7606930732727051 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.6345593333244324 $ _timestamp‚1.784367607235531e+09 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6462283134460449 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9957683086395264 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14416667819023132 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8558579683303833 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9101465940475464 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5545833706855774 )loss/val_total_loss‚8.620866685439873 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996483564376831 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09112142026424408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1094556674361229 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2570221424102783 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6502059102058411 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18876683712005615 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987663745880127 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6082642078399658 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7312976121902466 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 jet/accuracy_3_of_5‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46985912322998047 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.760745644569397 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5840864181518555 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6408697962760925 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49589696526527405 particle/accuracy_2_of_5‚1 _runtime‚4885.742167128 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8368104100227356 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24828195571899414 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7946666479110718 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.27331215143203735 *jet/accuracy_3_of_3‚0.0767824798822403 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29643386602401733 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17224976420402527 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7377908825874329 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6420308351516724 +jet/accuracy_1_of_1‚0.35267025232315063 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6764509081840515 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4620387554168701 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7078264951705933 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8998740911483765 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6824098825454712 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10390318930149078 +jet/accuracy_3_of_4‚0.08877447992563248 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7355902194976807 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24759581685066223 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8497544527053833 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 jet/accuracy_4_of_5‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6235604286193848 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22606870532035828 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7679516077041626 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6424446105957031 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6259852647781372 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2856338918209076 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1580902338027954 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6475377082824707 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31586146354675293 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2764119505882263 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08509891480207443 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7524863481521606 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1030573844909668 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.645194411277771 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9492474794387817 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25891080498695374 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9391651153564453 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8701770305633545 7"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.72503262758255 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_4‚0.9671691656112671 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 *jet/accuracy_2_of_3‚0.5633708238601685 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5584731101989746 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3877018392086029 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6327316761016846 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7262282371520996 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0702979564666748 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 epoch‚13 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17431291937828064 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8601285815238953 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9247224926948547 /particle/accuracy_2_of_2‚0.9998962879180908 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6839922070503235 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6865627765655518 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05389911308884621 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.041851580142974854 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970179796218872 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6174413561820984 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2591405510902405 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7validation_average_jet_accuracy‚0.39066410064697266 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9874575734138489 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2872249186038971 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21634553372859955 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5613088011741638 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46969544887542725 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_3‚1 trainer/global_step‚5599 _step‚461 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8779480457305908 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 particle/accuracy_2_of_3‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9908225536346436 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6815762519836426 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04489597678184509 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.684576153755188 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7777777910232544 *jet/accuracy_2_of_4‚0.7805314660072327 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963895678520203 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987130165100098 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6475377082824707 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09112142026424408 (particle/accuracy‚0.8472880125045776Í‹*P‘³‚(:† û“íÒªîŠð gpu.0.gpu‚85.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ×ãÏ´‚(:Ä û“íÒ¾µ»ò disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd0.in‚0! disk.nvme0n1.in‚ 6710.12109375 network.recv‚ 1.310778e+06 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 106.41796875 disk.drbd2165.in‚0( proc.memory.availableMB‚ 742.87109375 disk.rbd1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚10390.71337890625 proc.cpu.threads‚17 disk.rbd1.out‚12.5625) proc.memory.percent‚17.12479591369629 network.sent‚ 9.003859e+06$ disk./.usageGB‚589.9271011352539# disk.nvme0c0n1.in‚ 6710.12109375 disk.dm-1.in‚0 cpu‚26.04761183664232 disk.sda2.out‚0 disk.dm-1.out‚ 106.41796875 disk.dm-0.in‚ 6710.12109375( disk.nvme0c0n1.out‚10390.71337890625 disk.sda3.out‚ 106.41796875 disk.drbd2165.out‚0& disk.nvme0n1.out‚10390.71337890625$ memory_percent‚95.46587467193604" proc.memory.rssMB‚ 2805.7265625) disk./.usagePercent‚39.59050273899389 disk.sda.in‚0ü™n7“µ‚(:ˆ Š”íÒÁº„í gpu.0.gpu‚99.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í +u¹¶‚(:® Š”íÒ¯–Ÿï network.recv‚ 1.31436e+06# disk.nvme0c0n1.out‚ 10396.671875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.out‚ 10396.671875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 106.796875 network.sent‚ 9.060164e+06* disk./.usagePercent‚39.590515027355536 disk.drbd2165.in‚0 disk.dm-0.out‚ 10396.671875 disk.sda.out‚ 106.796875" proc.memory.rssMB‚ 2805.7265625 disk.sda2.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 743.1796875$ disk./.usageGB‚589.9272842407227 disk.dm-0.in‚ 6710.12109375 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 6710.12109375 disk.sda3.in‚0) proc.memory.percent‚17.12479591369629 proc.cpu.threads‚17! disk.nvme0n1.in‚ 6710.12109375 disk.sda3.out‚ 106.796875 disk.sda.in‚0$ memory_percent‚95.46399116516113 disk.drbd2165.out‚0 disk.rbd1.out‚12.625 cpu‚26.048516718356307‰7ŠŸ·‚(:’ ™”íÒÎ¥†í gpu.0.gpu‚87.0 gpu.0.memory‚25+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Êv?ĸ‚(:¹ ™”íÒìì‹ï disk.sda.out‚ 107.1484375$ memory_percent‚95.46470642089844 cpu‚26.049331819159963 network.recv‚ 1.317847e+06 disk.sda1.out‚0! disk.nvme0n1.in‚ 6710.12109375 proc.cpu.threads‚17 disk.sda1.in‚0# disk./.usageGB‚589.926326751709 disk.sda.in‚0 disk.sda3.out‚ 107.1484375 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 9.064998e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 6710.12109375" disk.dm-0.out‚10645.1025390625 disk.rbd1.in‚0$ proc.memory.availableMB‚743.0625 disk.dm-1.out‚ 107.1484375# disk.nvme0c0n1.in‚ 6710.12109375 disk.sda3.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚10645.1025390625 disk.rbd1.out‚12.625" proc.memory.rssMB‚ 2805.7265625) proc.memory.percent‚17.12479591369629 disk.dm-1.in‚0) disk./.usagePercent‚39.59045076946444' disk.nvme0c0n1.out‚10645.1025390625™C¢î2¹- _wandbruntime‚4928  _runtime‚4928ȵ±ÊǺÁ & lr-AdamW/pg2‚0.0004849605550286481 trainer/global_step‚5649 % _timestamp‚1.7843676501947904e+09 _runtime‚4928.700760982 _step‚462 & lr-AdamW/pg1‚0.0004849605550286481ªZ«#Ì»Æ _step‚462 & lr-AdamW/pg1‚0.0004849605550286481 & lr-AdamW/pg2‚0.0004849605550286481 trainer/global_step‚5649 % _timestamp‚1.7843676501947904e+09 _runtime‚4928.700760982ÎÚ 12¼- _wandbruntime‚4928  _runtime‚4928pâ'½ÜO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44813984 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6039499767808 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148403 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52334419 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52344396 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚858961591508992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚858961574469632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189219233957888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚858961574469632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚854541847851008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5977788924928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195258733725696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚860510177665460 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5977788924928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5977447585792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813986 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5977447585792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22061561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6039409093632 _step‚463 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚864939380433920 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022055424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4928.705733532 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9900408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚858961591508992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚864939380433920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022055424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚860510535254710 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843676501958265e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813986 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158382 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9900408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚854541864890368 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚858961591508992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚864939380433920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚864939022055424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚860510535254710 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843676501958265e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44813986 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97158382 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52334419 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52344396 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚858961591508992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚858961574469632 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189219233957888 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44813984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚858961574469632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚854541847851008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5977788924928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195258733725696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚860510177665460 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5977788924928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5977447585792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44813986 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5977447585792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22061561 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚90674176 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6039409093632 _step‚463 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚864939380433920 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚864939022055424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4928.705733532 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12163482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5649 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97165087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚4929.333783678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865087634097664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12163235 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978460212736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9902156 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52353462 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5978801559040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195293078697472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189252644104192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969681418176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22065388 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44821616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22065638 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44821618 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865087275712000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5978460212736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52343471 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6040434593280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5969340868174 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚860657699391054 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843676508276162e+09ž¸a,ñ&ÁëM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859108815499264 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚854688358522880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859108815499264 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚86472704 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52343471 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195293183782400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44821616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚865087634097664 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12163235 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5978460212736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9902156 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52353462 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5978801559040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚854688375562240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195293078697472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189252644104192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5969681418176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22065388 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 105084928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44821616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22065638 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44821618 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚865087275712000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5978460212736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52343471 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6040434593280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5969340868174 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚860657699391054 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843676508276162e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12163482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚5649 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97165087 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚4929.333783678 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22253884 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚866323625951232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9987661 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872351769038848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6090834053120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861867398483968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769038848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872352127417344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 trainer/global_step‚5699 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190890671884288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52798848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12265963 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚866323625951232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143087616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45195584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867886685335218 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6018946301360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22253884 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚866323625951232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9987661 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190890653272064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚872351769038848 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9987658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6090834053120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚861867398483968 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚872351769038848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚872352127417344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52798848 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 trainer/global_step‚5699 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190890671884288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52798848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12265963 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚866323625951232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6028143087616 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45195584 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚867886685335218 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6018946301360 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚468 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197015935791616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52797923 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚862013892116480 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98001139 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45203216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872500381081088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9989403 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚866470884020224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12268047 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197016042973696 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52807914 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚866470884020224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190924082030592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚872500022695424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9989406 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12268316 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22257450 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784367692352678e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52797923 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚872500381081088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6029497060864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6091853761024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚866470866980864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6020297905084 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚868033849471562 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚866470866980864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52807914 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4970.858825097 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45203218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6029497060864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060924ÕWÙ ÓêM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197016042973696 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52807914 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚866470884020224 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190924082030592 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚872500022695424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9989406 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12268316 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22257450 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 $ _timestamp‚1.784367692352678e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52797923 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚872500381081088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6029497060864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6091853761024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚866470866980864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6020297905084 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98011132 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚88569856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚868033849471562 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚866470866980864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45203218 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52807914 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4970.858825097 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45203218 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6029497060864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚868034207060924 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001139 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚468 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197015935791616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52797923 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚862013909155840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚862013892116480 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98001139 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45203216 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚872500381081088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9989403 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45203216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚866470884020224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12268047 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438Ôw_Ø2Ô- _wandbruntime‚4970  _runtime‚49702U!cÕ€ 2loss/SRqqt1/assignment_loss‚0.5168683355641447 2loss/SRqqt2/assignment_loss‚0.5050653890189606 1loss/FBt2/assignment_loss‚0.03867938278923702 /loss/FRt1/detection_loss‚0.5804041768821795 epoch‚14 _runtime‚4970.859967579 2loss/FBt1/assignment_loss‚0.013670625563789524 %loss/total_loss‚8.660486057414605 trainer/global_step‚5699 0loss/FRt2/assignment_loss‚0.8543733563352738 .loss/FRt2/detection_loss‚1.431281773592316 0loss/SRqqt1/detection_loss‚1.055830826932514 1loss/SRqqt2/detection_loss‚0.6711866732101236 /loss/FBt1/detection_loss‚0.4659955522542832 /loss/FBt2/detection_loss‚0.7225947380065918 0loss/FRt1/assignment_loss‚1.8045352272651902 $ _timestamp‚1.784367692354287e+09 _step‚469«ˆ‰# Ö… 2loss/SRqqt1/assignment_loss‚0.5168683355641447 2loss/SRqqt2/assignment_loss‚0.5050653890189606 1loss/FBt2/assignment_loss‚0.03867938278923702 /loss/FRt1/detection_loss‚0.5804041768821795 epoch‚14 _runtime‚4970.859967579 2loss/FBt1/assignment_loss‚0.013670625563789524 %loss/total_loss‚8.660486057414605 trainer/global_step‚5699 0loss/FRt2/assignment_loss‚0.8543733563352738 .loss/FRt2/detection_loss‚1.431281773592316 0loss/SRqqt1/detection_loss‚1.055830826932514 1loss/SRqqt2/detection_loss‚0.6711866732101236 /loss/FBt1/detection_loss‚0.4659955522542832 /loss/FBt2/detection_loss‚0.7225947380065918 0loss/FRt1/assignment_loss‚1.8045352272651902 $ _timestamp‚1.784367692354287e+09 _step‚469Õ‚‘(”“ׂ(:ˆ Õ”íÒÓØí gpu.0.gpu‚87.0 gpu.0.memory‚26+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é½wÅØ‚(:º Õ”íÒ˜ìÖï disk.drbd2165.in‚0 cpu‚26.052580433206227 disk.dm-1.in‚0 disk.dm-1.out‚ 108.640625# disk.dm-0.out‚10663.82568359375( disk.nvme0c0n1.out‚10663.82568359375 disk.nvme0n1.in‚ 6810.078125$ memory_percent‚95.45724391937256 disk.sda1.out‚0 network.recv‚ 1.332799e+06* disk./.usagePercent‚39.590577493193884 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 6810.078125 disk.rbd0.in‚0) proc.memory.percent‚17.12479591369629 network.sent‚ 9.143054e+06 disk.rbd1.in‚0 disk.rbd1.out‚12.8125 disk.sda3.out‚ 108.640625 disk.dm-0.in‚ 6810.078125( proc.memory.availableMB‚ 744.28515625 proc.cpu.threads‚17 disk.sda2.out‚0" proc.memory.rssMB‚ 2805.7265625 disk.sda.out‚ 108.640625 disk.sda3.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚10663.82568359375$ disk./.usageGB‚589.9282150268555 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd0.out‚0ü|1§Ù‚(:’ ä”íÒ‹Ž¨í gpu.0.gpu‚86.0 gpu.0.memory‚27+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚262.071' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è» ÅÚ‚(:º ä”íÒ€”Ôï disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 6935.03515625 disk.sda1.in‚0 network.recv‚ 1.337598e+06 disk.dm-1.in‚0( disk.nvme0c0n1.out‚10668.36572265625 disk.sda.in‚0 network.sent‚ 9.205855e+06 disk.rbd1.out‚12.875" proc.memory.rssMB‚ 2805.7265625 cpu‚26.053419514247167) disk./.usagePercent‚39.59055726859868 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 109.21875 disk.sda1.out‚0 disk.sda3.out‚ 109.21875 disk.drbd2165.in‚0 disk.sda3.in‚0$ memory_percent‚95.45683860778809 disk.dm-1.out‚ 109.21875 disk.rbd0.out‚0$ disk./.usageGB‚589.9279136657715 disk.dm-0.in‚ 6935.03515625# disk.dm-0.out‚10668.36572265625' proc.memory.availableMB‚ 744.3515625) proc.memory.percent‚17.12479591369629 proc.cpu.threads‚17 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 6935.03515625& disk.nvme0n1.out‚10668.36572265625¦e>‹žÛ‚(:“ ó”íҮьí gpu.0.gpu‚86.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚247.021' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.404199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cS‰2ÃÜ‚(:¸ ó”íÒÈòÑï' proc.memory.availableMB‚ 743.7421875# disk.nvme0c0n1.in‚ 6935.03515625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 109.6328125 disk.drbd2165.out‚0! disk.dm-0.out‚10670.818359375 network.recv‚ 1.341067e+06) disk./.usagePercent‚39.59056085270416& disk.nvme0c0n1.out‚10670.818359375! disk.nvme0n1.in‚ 6935.03515625 disk.sda.out‚ 109.6328125 disk.sda3.in‚0$ memory_percent‚95.46055793762207 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 9.21079e+06 disk.sda.in‚0 disk.sda3.out‚ 109.6328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 6935.03515625 cpu‚26.05430329674829$ disk.nvme0n1.out‚10670.818359375 disk.sda2.in‚0" proc.memory.rssMB‚ 2805.7265625) proc.memory.percent‚17.12479591369629$ disk./.usageGB‚589.9279670715332 disk.rbd1.in‚0 disk.rbd1.out‚12.875 disk.sda1.out‚0 disk.drbd2165.in‚0@2Ý- _wandbruntime‚5011  _runtime‚5011gÓ;¢ÈÞ _runtime‚5011.913767964 _step‚470 ' lr-AdamW/pg1‚0.00048145825158978095 ' lr-AdamW/pg2‚0.00048145825158978095 trainer/global_step‚5749 $ _timestamp‚1.784367733407804e+09RòÕ@ÍßÇ _step‚470 ' lr-AdamW/pg1‚0.00048145825158978095 ' lr-AdamW/pg2‚0.00048145825158978095 trainer/global_step‚5749 $ _timestamp‚1.784367733407804e+09 _runtime‚5011.913767964ÖPÖè2à- _wandbruntime‚5011  _runtime‚5011·^)§ã'áÝO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98820507 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45577184 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192562091198464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6069562788268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10074911 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879764874400768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6078838589440 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 trainer/global_step‚5749 $ _timestamp‚1.784367733408893e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚873685694472192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚873685694472192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6142344307712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6079179928576 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53253300 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _step‚471 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22445934 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6079179928576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12371291 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45577184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53243323 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚5011.915019387 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830486 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45577186 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53253300 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516022272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12371026 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10074908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879764874400768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875262835415726 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚873685677432832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879764516022272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚869192915038208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6069903338158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198704454118400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198704349026304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6078838589440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875262477826476 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53243323 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6142257827840 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879764516022272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12371026 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10074908 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879764874400768 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875262835415726 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚873685677432832 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚879764516022272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45577186 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚869192915038208 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6069903338158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198704454118400 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198704349026304 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6078838589440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875262477826476 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53243323 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6142257827840 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192562091198464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6069562788268 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10074911 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚879764874400768 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6078838589440 A8DeviceStatsMonitor.on_tra'\ÉZ©in_batch_start/segment.all.peak‚1438 trainer/global_step‚5749 $ _timestamp‚1.784367733408893e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚873685694472192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169×T½Ñx2ã- _wandbruntime‚5012  _runtime‚50124 Gî&äèM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192595538569216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚5012.545335908 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192562688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53262366 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843677340392294e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45584816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837191 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10076653 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198738791885312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚873832918462464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6080192562688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873832935501824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12373106 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875409999552070 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚869339425710080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879913128064512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6143362595328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198738901164544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851216384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875410357141432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45584818 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45584818 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22450035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12373379 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769678848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22449759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10076656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6070573841990 _step‚472 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53252375 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚5749 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 ¦ê=ó&åíM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53262366 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6143271928320 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45584816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98837191 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚873832935501824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10076653 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198738791885312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚873832918462464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6080192562688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚873832935501824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12373106 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875409999552070 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚869339425710080 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879913128064512 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚90667008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚869339442749440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6143362595328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚873832918462464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198738901164544 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6079851216384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192595519956992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875410357141432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45584818 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45584818 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22450035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12373379 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879912769678848 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22449759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10076656 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6070573841990 _step‚472 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53252375 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 109279232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 trainer/global_step‚5749 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192595538569216 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 _runtime‚5012.545335908 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6080192562688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53262366 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 % _timestamp‚1.7843677340392294e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666538 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6193651710976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876518448631808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12475941 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10162161 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚ 5053.56674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚881047745953792 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚475 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53707752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚5799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263005696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 % _timestamp‚1.7843677750569675e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53697775 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534091264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6193571522560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚887177621384192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882638627906984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876518465671168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887177263005696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825066 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6129875430400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceS¿÷=tatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6129875430400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200427100647424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45958786 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53697775 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚881047728914432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22638102 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12475696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6129534091264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚881047745953792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚882638985496234 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚881047728914432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10162158 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194233547737088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 trainer/global_step‚5799 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887177263005696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 % _timestamp‚1.7843677750569675e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53697775 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6129534091264 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6193571522560 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚887177621384192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚882638627906984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚876518465671168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚887177263005696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6120519825066 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6129875430400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6129875430400 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200427100647424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚80188416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45958786 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53697775 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚881047728914432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22638102 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12475696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6129534091264 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚881047745953792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚882638985496234 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚881047728914432 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10162158 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194233547737088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666538 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6193651710976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚876518448631808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12475941 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10162161 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _runtime‚ 5053.56674624 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚881047745953792 UHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚98800640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 _step‚475Ûv˜»û2ó- _wandbruntime‚5054  _runtime‚5054㜮í&ôçM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6130888064512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚881194969944064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6130546718208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194266957883392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚5799 % _timestamp‚1.7843677756879835e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200461556431360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328898 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22641624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6121530878900 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10163906 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚476 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45966418 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10163903 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚ 5054.19407664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚887325875047936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53716818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6194598547968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45966416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12477721 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882786507221940 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12477969 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53706827 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚887325875047936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45966418 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882786149632578 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6130546718208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6194682923520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚887325516662272W‹¸!õìM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12477969 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53706827 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚84375552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚887325875047936 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45966418 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚882786149632578 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6130546718208 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6194682923520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚887325516662272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6130888064512 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚881194969944064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6130546718208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 102987776 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194266957883392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 trainer/global_step‚5799 % _timestamp‚1.7843677756879835e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200461556431360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚876664959303680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6121190328898 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22641624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6121530878900 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10163906 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚476 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45966418 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10163903 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 _runtime‚ 5054.19407664 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚887325875047936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53716818 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6194598ÏK S:547968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45966416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12477721 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚882786507221940 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6244732770816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10249411 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229593088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576742 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843678165128176e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _step‚479 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46340386 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229593088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009989120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570932224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202149811205632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚894590368367616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888409780396032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195904985663488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12579517 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6180570932224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6244825542144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚5095.019142698 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195904967051264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚5849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890014777987492 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚883843982225408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894590368367616 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54162204 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54162204 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888409797435392Ö/ í'„çO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54162204 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚888409797435392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46340384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54152227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚883843999264768 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6244732770816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10249411 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6180229593088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890015135576742 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 % _timestamp‚1.7843678165128176e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888409780396032 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 _step‚479 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46340386 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6180229593088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚894590009989120 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9977 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6180570932224 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202149811205632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚894590368367616 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888409780396032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195904985663488 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12579517 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6180570932224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6244825542144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 _runtime‚5095.019142698 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195904967051264 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 trainer/global_step‚5849 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890014777987492 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚883843982225408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚894590368367616 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54162204 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169ߢÊ;§2…- _wandbruntime‚5095  _runtime‚5095†¥púø†ëM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10251153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202184260639744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888557038465024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263645696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46348016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54161279 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6172147365808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894738622031360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6181242220032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚883990509936640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6171806815806 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100519288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46348018 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202184149263360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46348018 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22832731 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888557021425664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6181583566336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12581578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10251156 _runtime‚5095.650798019 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888557038465024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚890162299713086 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46348016 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195938414422016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5849 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6245753453568 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54161279 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169­#›oö&‡ðM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100519288 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46348018 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚269 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚202184149263360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888557021425664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46348018 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22832731 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888557021425664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6181583566336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12581578 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10251156 _runtime‚5095.650798019 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888557038465024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚890162299713086 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46348016 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195938414422016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚5849 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 341346304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 _step‚480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6245753453568 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54161279 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10251153 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202184260639744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚883990492897280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888557038465024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚894738263645696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46348016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54161279 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6172147365808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚894738622031360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6181242220032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚883990509936640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6171806815806 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438à >wÙ2ˆ- _wandbruntime‚5095  _runtime‚5095«­î ‰ƒ 2loss/SRqqt1/assignment_loss‚0.5763572136887255 2loss/FBt1/assignment_loss‚0.056284319927402335 /loss/FRt1/detection_loss‚0.5618406272401676 % _timestamp‚1.7843678171460967e+09 _runtime‚5095.651794531 _step‚481 0loss/FRt1/assignment_loss‚1.8640740072814037 trainer/global_step‚5849 epoch‚14 1loss/SRqqt2/assignment_loss‚0.526906544043307 /loss/FRt2/detection_loss‚1.4183820509016694 1loss/SRqqt2/detection_loss‚0.6700192762963364 0loss/FBt1/detection_loss‚0.41670474311686123 /loss/FBt2/detection_loss‚0.8108785106076135 0loss/FRt2/assignment_loss‚0.9373329755848151 2loss/FBt2/assignment_loss‚0.011011952761721962 0loss/SRqqt1/detection_loss‚1.009522859701615 %loss/total_loss‚8.859315081151639}Â|`Šˆ epoch‚14 1loss/SRqqt2/assignment_loss‚0.526906544043307 /loss/FRt2/detection_loss‚1.4183820509016694 1loss/SRqqt2/detection_loss‚0.6700192762963364 0loss/FBt1/detection_loss‚0.41670474311686123 /loss/FBt2/detection_loss‚0.8108785106076135 0loss/FRt2/assignment_loss‚0.9373329755848151 2loss/FBt2/assignment_loss‚0.011011952761721962 0loss/SRqqt1/detection_loss‚1.009522859701615 %loss/total_loss‚8.859315081151639 2loss/SRqqt1/assignment_loss‚0.5763572136887255 2loss/FBt1/assignment_loss‚0.056284319927402335 /loss/FRt1/detection_loss‚0.5618406272401676 % _timestamp‚1.7843678171460967e+09 _runtime‚5095.651794531 _step‚481 0loss/FRt1/assignment_loss‚1.8640740072814037 trainer/global_step‚5849áa`C“‹‚(:ˆ Í•íÒœª–î gpu.0.gpu‚96.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^=øËŒ‚(:À Í•íÒøÀüï disk.rbd0.out‚0% disk.nvme0n1.out‚10695.7431640625 disk.sda2.in‚0( proc.memory.availableMB‚ 743.73828125 cpu‚26.059346369965194 disk.sda3.out‚ 111.30859375" disk.nvme0c0n1.in‚ 7030.3984375" proc.memory.rssMB‚ 2805.7265625 proc.cpu.threads‚17$ disk./.usageGB‚589.9278907775879 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚95.46058177947998 network.sent‚ 9.356355e+06 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 7030.3984375 disk.dm-0.in‚ 7030.3984375' disk.nvme0c0n1.out‚10695.7431640625 disk.sda1.in‚0* disk./.usagePercent‚39.590555732553476 disk.rbd1.in‚0 disk.rbd1.out‚13.0625) proc.memory.percent‚17.12479591369629 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚10695.7431640625 disk.sda.in‚0 disk.sda.out‚ 111.30859375 disk.dm-1.out‚ 111.30859375 network.recv‚ 1.363707e+06 disk.sda3.in‚0z¤Gì“‚(:ˆ Ü•íҟ™í gpu.0.gpu‚69.0 gpu.0.memory‚24+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¾“Ô»Ž‚(:° Ü•íүüï disk.dm-0.out‚ 10698.2265625 disk.dm-1.in‚0 disk.dm-1.out‚ 111.46875" proc.memory.rssMB‚ 2805.7265625) proc.memory.percent‚17.12479591369629 cpu‚26.060248667938485 disk.dm-0.in‚ 7101.16796875 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 743.79296875 disk.sda1.out‚0* disk./.usagePercent‚39.590568020915114 disk.sda3.in‚0 disk.sda3.out‚ 111.46875 proc.cpu.threads‚17$ disk./.usageGB‚589.9280738830566 disk.rbd1.out‚13.1875# disk.nvme0c0n1.in‚ 7101.16796875 network.sent‚ 9.41914e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 111.46875 network.recv‚ 1.368121e+06 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7101.16796875" disk.nvme0n1.out‚ 10698.2265625$ disk.nvme0c0n1.out‚ 10698.2265625 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚95.46024799346924 disk.rbd0.out‚0ñ’Dž‚(:“ ë•íÒÄÙòí gpu.0.gpu‚45.0 gpu.0.memory‚16+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚134.552' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.910400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðûY©Í‚(: ë•íÒŠã…ï proc.cpu.threads‚17 disk.sda1.in‚0" proc.memory.rssMB‚ 2805.7265625 disk.rbd0.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 744.02734375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 111.83984375) proc.memory.percent‚17.12479591369629) disk./.usagePercent‚39.59057211703566 disk.rbd0.in‚0! disk.nvme0n1.in‚ 7101.16796875% disk.nvme0n1.out‚10702.7939453125 disk.dm-0.in‚ 7101.16796875 disk.sda.out‚ 111.83984375 disk.drbd2165.out‚0$ memory_percent‚95.45881748199463 network.sent‚ 9.424237e+06$ disk./.usageGB‚589.9281349182129 disk.sda3.out‚ 111.83984375 network.recv‚ 1.371481e+06 disk.rbd1.out‚13.1875# disk.nvme0c0n1.in‚ 7101.16796875' disk.nvme0c0n1.out‚10702.7939453125 disk.sda3.in‚0 disk.drbd2165.in‚0" disk.dm-0.out‚10702.7939453125 disk.rbd1.in‚0 disk.dm-1.in‚0 cpu‚26.061144748153755VRät2‘- _wandbruntime‚5136  _runtime‚5136¤ŒÑÇ’Á & lr-AdamW/pg1‚0.0004761133421648045 & lr-AdamW/pg2‚0.0004761133421648045 trainer/global_step‚5899 % _timestamp‚1.7843678581688015e+09 _runtime‚5136.674765334 _step‚482¢P„ƒÌ“Æ _runtime‚5136.674765334 _step‚482 & lr-AdamW/pg1‚0.0004761133421648045 & lr-AdamW/pg2‚0.0004761133421648045 trainer/global_step‚5899 % _timestamp‚1.7843678581688015e+09âÊËO«2”- _wandbruntime‚5136  _runtime‚5136]÷‚è'•âO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚203872310184960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 111383552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54606679 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚895771848916992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚275 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46721986 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6230925094912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚891169515819008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10336658 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚891169532858368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚5136.676023429 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54606679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328663 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231266434048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46721986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46721984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203872421568512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101338642 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197576404977664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115351040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6230925094912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115351040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10336661 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266434048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6295997978624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚895771831877632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798882 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248992 trainer/global_step‚5899 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12683546 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23020207 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54616656 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚895771831877632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12683271 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197576423589888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6295905207296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚483 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756972544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23019929 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚897390928068000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚891169532858368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 _runtime‚5136.676023429 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚92771328 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54606679 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328663 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231266434048 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46721986 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46721984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚203872421568512 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101338642 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197576404977664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚902003115351040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6230925094912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚902003115351040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10336661 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46721984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6231266434048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6295997978624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚895771831877632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6221752798882 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221412248992 trainer/global_step‚5899 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12683546 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23020207 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚895771848916992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54616656 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚895771831877632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12683271 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197576423589888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6295905207296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚483 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚902002756972544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23019929ãn·¶ó2—- _wandbruntime‚5137  _runtime‚5137Æ“Ò ò&˜ìM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23023717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10338406 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222423302714 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚895919072907264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895919089946624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203906757377536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54615731 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6231937721856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6222763852716 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚895919072907264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54625722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚891316026490880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚891316043530240 _runtime‚5137.306966892 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 trainer/global_step‚5899 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902151369014784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6231937721856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897538807382956 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897538449793594 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625722 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197609852348416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6297020599808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6296923641344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46729618 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843678588007793e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚895919089946624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197609833736192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232279068160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12685584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232279068160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576Ì¢ô÷&™ñM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232279068160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23023717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10338406 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222423302714 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚895919072907264 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 115570688 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1169 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46729618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚895919089946624 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203906757377536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚96958464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54615731 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6231937721856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6222763852716 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚895919072907264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54625722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚891316026490880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚891316043530240 _runtime‚5137.306966892 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 trainer/global_step‚5899 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚902151369014784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6231937721856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚897538807382956 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚897538449793594 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54625722 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197609852348416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6297020599808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6296923641344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46729618 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843678588007793e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚895919089946624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197609833736192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232279068160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12685584 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0äìCú2š- _wandbruntime‚5137  _runtime‚5137: ° ›ƒ %loss/total_loss‚9.484000400744964 /loss/FRt1/assignment_loss‚1.810075738424988 0loss/FRt2/assignment_loss‚0.9418864667159805 2loss/SRqqt2/assignment_loss‚0.5608722218747159 /loss/FBt1/detection_loss‚0.5557014449337006 % _timestamp‚1.7843678588023274e+09 2loss/FBt1/assignment_loss‚0.028943167675509654 3loss/FBt2/assignment_loss‚0.0010091709275927535 _step‚485 /loss/FBt2/detection_loss‚1.2386445379334277 epoch‚14 2loss/SRqqt1/assignment_loss‚0.6949250860188234 /loss/FRt2/detection_loss‚1.4137775134964738 0loss/SRqqt1/detection_loss‚1.052852988325148 1loss/SRqqt2/detection_loss‚0.6282053922325057 trainer/global_step‚5899 _runtime‚5137.308018156 /loss/FRt1/detection_loss‚0.5571066721860979r”†Xœˆ %loss/total_loss‚9.484000400744964 /loss/FRt1/assignment_loss‚1.810075738424988 0loss/FRt2/assignment_loss‚0.9418864667159805 2loss/SRqqt2/assignment_loss‚0.5608722218747159 /loss/FBt1/detection_loss‚0.5557014449337006 % _timestamp‚1.7843678588023274e+09 2loss/FBt1/assignment_loss‚0.028943167675509654 3loss/FBt2/assignment_loss‚0.0010091709275927535 _step‚485 /loss/FBt2/detection_loss‚1.2386445379334277 epoch‚14 2loss/SRqqt1/assignment_loss‚0.6949250860188234 /loss/FRt2/detection_loss‚1.4137775134964738 0loss/SRqqt1/detection_loss‚1.052852988325148 1loss/SRqqt2/detection_loss‚0.6282053922325057 trainer/global_step‚5899 _runtime‚5137.308018156 /loss/FRt1/detection_loss‚0.5571066721860979åPº“‚(:ˆ ú•íÒ·ž™í gpu.0.gpu‚95.0 gpu.0.memory‚34+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚36.0 gpu.0.powerWatts‚276.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<÷8(Àž‚(:µ ú•íÒâ´µñ& proc.memory.availableMB‚ 743.796875 disk.dm-1.out‚ 112.0859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 7101.16796875 disk.rbd1.in‚0 cpu‚26.061812845433643 network.sent‚ 9.429254e+06 disk.sda1.out‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 112.0859375# disk.nvme0n1.out‚10707.39453125# disk.nvme0c0n1.in‚ 7101.16796875$ memory_percent‚95.46022415161133 network.recv‚ 1.374866e+06) disk./.usagePercent‚39.59057672517128 disk.sda.in‚0 disk.sda2.out‚0 disk.rbd1.out‚13.25$ disk./.usageGB‚589.9282035827637 disk.drbd2165.out‚0 disk.dm-0.out‚10707.39453125 proc.cpu.threads‚17% disk.nvme0c0n1.out‚10707.39453125! disk.nvme0n1.in‚ 7101.16796875 disk.sda.out‚ 112.0859375 disk.sda1.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2805.7265625) proc.memory.percent‚17.12479591369629ÝÏKüqŸ‚(:‘ ‰–íÒÜ‚ï gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErro4 ~(+rs‚0# gpu.0.uncorrectedMemoryErrors‚0]©jZÆ ‚(:» ‰–íÒħ†ð$ disk.nvme0n1.out‚10714.978515625 disk.sda2.in‚0" proc.memory.rssMB‚ 2805.7265625 disk.sda3.out‚ 112.40234375 disk.sda2.out‚0& disk.nvme0c0n1.out‚10714.978515625$ memory_percent‚95.46034336090088 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 743.77734375) proc.memory.percent‚17.12479591369629 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 7101.16796875 network.recv‚ 1.379732e+06 disk.drbd2165.out‚0 disk.rbd0.in‚0! disk.dm-0.out‚10714.978515625 disk.sda3.in‚0 disk.dm-1.out‚ 112.40234375 disk.rbd1.in‚0 disk.rbd1.out‚13.25 disk.sda.out‚ 112.40234375 cpu‚26.06270914719786) disk./.usagePercent‚39.59051937948361 disk.sda1.out‚0 disk.dm-0.in‚ 7101.16796875 disk.sda.in‚0 network.sent‚ 9.49407e+06$ disk./.usageGB‚589.9273490905762 disk.rbd0.out‚0! disk.nvme0n1.in‚ 7101.16796875hõL“¡‚(:ˆ ˜–íÒ˜ýñí gpu.0.gpu‚50.0 gpu.0.memory‚17+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚35.0 gpu.0.powerWatts‚108.231' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.6462 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LDžìÀ¢‚(:µ ˜–íÒ«›ðï$ disk./.usageGB‚589.9274215698242 disk.sda1.in‚0 disk.sda3.out‚ 112.7578125 disk.nvme0n1.in‚ 7194.09375 cpu‚26.063443459233476 network.sent‚ 9.49715e+06 disk.sda1.out‚0 disk.dm-1.out‚ 112.7578125 disk.rbd0.in‚0$ memory_percent‚96.96178436279297% proc.memory.availableMB‚ 497.78125) disk./.usagePercent‚39.59052424362676 disk.sda3.in‚0# disk.dm-0.out‚10719.28369140625& disk.nvme0n1.out‚10719.28369140625 disk.sda2.in‚0( disk.nvme0c0n1.out‚10719.28369140625 disk.drbd2165.out‚0) proc.memory.percent‚17.12479591369629 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 7194.09375 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚13.3125 disk.rbd0.out‚0 network.recv‚ 1.383048e+06 disk.sda.out‚ 112.7578125 disk.drbd2165.in‚0 disk.dm-0.in‚ 7194.09375" proc.memory.rssMB‚ 2805.7265625 proc.cpu.threads‚17(ÏAK2£- _wandbruntime‚5178  _runtime‚5178‡ °lǤÁ _step‚486 & lr-AdamW/pg1‚0.0004743077748312799 & lr-AdamW/pg2‚0.0004743077748312799 trainer/global_step‚5949 % _timestamp‚1.7843678996242893e+09 _runtime‚5178.130302021:´Z¥Ì¥Æ & lr-AdamW/pg1‚0.0004743077748312799 & lr-AdamW/pg2‚0.0004743077748312799 trainer/global_step‚5949 % _timestamp‚1.7843678996242893e+09 _runtime‚5178.130302021 _step‚486æ´Vê”2¦- _wandbruntime‚5178  _runtime‚5178Aov{æ'§àO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47103584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071108 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6281620596736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55061131 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚898495049412608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164715 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚909415862334464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205594964680192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23211414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚5178.131667779 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903133900398592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903133900398592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10423911 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503955968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5949 # _timestamp‚1.78436789962543e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23211155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6272028735900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚898495066451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6272369285790 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚903133883359232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12787503 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚903133883359232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚487 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6347103163904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10423908 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12787247 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904767078148508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚909415862334464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205594859588096Xýýqë'¨åO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6347103163904 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10423908 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12787247 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚904767078148508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚909415862334464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205594859588096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47103584 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55071108 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6281620596736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55061131 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚898495049412608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164715 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚909415862334464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205594964680192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚86479872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47103584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23211414 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 23842521088 _runtime‚5178.131667779 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚903133900398592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚903133900398592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 554758144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10423911 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 105092096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚909415503955968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 trainer/global_step‚5949 # _timestamp‚1.78436789962543e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23211155 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6272028735900 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 341339136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 23190607872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚898495066451968 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 340549890 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12101 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚904767435737758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6272369285790 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚903133883359232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12787503 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚903133883359232 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 _step‚487 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0çQúg/2©- _wandbruntime‚5178  _runtime‚5178fø¢—n$ªìM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12789590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205629410677760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909564115998208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10425653 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚5949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55070183 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10425656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚904914957463464 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115998208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757612544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚903281124388864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚5178.765860228 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55070183 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205629299301376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23215246 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6348027638784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23214984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6348120402944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102181399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47111216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843679002597735e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47111218 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6273380339624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚903281124388864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6273039789622 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6282633223680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974569984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6282974569984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199281271662592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 M=Deƒ(º4„viceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6282633223680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55080174 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360¨G÷&«ñM H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 23190607872 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 _runtime‚5178.765860228 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55070183 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205629299301376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23215246 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚898641560084480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6348027638784 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23214984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6348120402944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102181399 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47111216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102181399 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47111216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚903281141428224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 24406654976 % _timestamp‚1.7843679002597735e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47111218 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 23025176576 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6273380339624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚903281124388864 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6273039789622 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6282633223680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚488 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6282974569984 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6282974569984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199281271662592 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚92764160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6282633223680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55080174 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12789590 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205629410677760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚909564115998208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 23702949376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10425653 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 trainer/global_step‚5949 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 554758144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55070183 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10425656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 111376384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚904914957463464 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚903281141428224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚909564115998208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚909563757612544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚903281124388864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976èÍcÌð2¬- _wandbruntime‚5178  _runtime‚5178­Ê€ ­„ 1loss/SRqqt1/detection_loss‚1.1871814890287051 /loss/FBt1/detection_loss‚0.6887558995715288 /loss/FBt2/detection_loss‚0.7630643514972745 epoch‚14 2loss/SRqqt2/assignment_loss‚0.5036313879835543 2loss/FBt1/assignment_loss‚0.014198720539356485 /loss/FRt1/detection_loss‚0.5718526592169609 trainer/global_step‚5949 _step‚489 0loss/FRt1/assignment_loss‚1.8945983594751612 /loss/FRt2/detection_loss‚1.3731732917078159 1loss/SRqqt2/detection_loss‚0.6013392736799045 %loss/total_loss‚9.037117893684227 % _timestamp‚1.7843679002612612e+09 0loss/FRt2/assignment_loss‚0.8711706671592997 _runtime‚5178.766953171 2loss/SRqqt1/assignment_loss‚0.5533187270829152 2loss/FBt2/assignment_loss‚0.0148330667417500348©Œã®‰ 0loss/FRt2/assignment_loss‚0.8711706671592997 _runtime‚5178.766953171 2loss/SRqqt1/assignment_loss‚0.5533187270829152 2loss/FBt2/assignment_loss‚0.014833066741750034 1loss/SRqqt1/detection_loss‚1.1871814890287051 /loss/FBt1/detection_loss‚0.6887558995715288 /loss/FBt2/detection_loss‚0.7630643514972745 epoch‚14 2loss/SRqqt2/assignment_loss‚0.5036313879835543 2loss/FBt1/assignment_loss‚0.014198720539356485 /loss/FRt1/detection_loss‚0.5718526592169609 trainer/global_step‚5949 _step‚489 0loss/FRt1/assignment_loss‚1.8945983594751612 /loss/FRt2/detection_loss‚1.3731732917078159 1loss/SRqqt2/detection_loss‚0.6013392736799045 %loss/total_loss‚9.037117893684227 % _timestamp‚1.7843679002612612e+09éI mP“¯‚(:ˆ §–íÒÙ£Öí gpu.0.gpu‚37.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÞTBð‚(:¸ §–íÒ˜õ³ï network.sent‚ 9.501986e+06 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚10726.80810546875( proc.memory.availableMB‚ 497.51171875) disk./.usagePercent‚39.59048737854183 disk.sda.out‚ 113.8671875 disk.sda3.in‚0 disk.dm-1.out‚ 113.8671875 disk.rbd0.in‚0) proc.memory.percent‚17.12479591369629# disk./.usageGB‚589.926872253418 disk.sda2.out‚0& disk.nvme0n1.out‚10726.80810546875 disk.rbd1.out‚13.3125 disk.sda.in‚0 disk.drbd2165.in‚0$ memory_percent‚96.96342945098877 cpu‚26.064272692594695" proc.memory.rssMB‚ 2805.7265625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 7194.09375 network.recv‚ 1.386618e+06 disk.nvme0c0n1.in‚ 7194.09375( disk.nvme0c0n1.out‚10726.80810546875 disk.sda3.out‚ 113.8671875 disk.drbd2165.out‚0 disk.dm-0.in‚ 7194.09375 proc.cpu.threads‚178ÞÞX±‚(:’ ¶–íÒ´‡Üì gpu.0.gpu‚37.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚37.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kï.^¼²‚(:± ¶–íÒÕ³Æï network.recv‚ 1.390931e+06 disk.nvme0n1.in‚ 7194.09375 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 497.48828125) proc.memory.percent‚17.12479591369629 disk.sda3.out‚ 115.234375 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚13.375 disk.sda1.out‚0 network.sent‚ 9.567021e+06' disk.nvme0c0n1.out‚10734.3896484375" proc.memory.rssMB‚ 2805.7265625 disk.dm-1.out‚ 115.234375 disk.nvme0c0n1.in‚ 7194.09375$ memory_percent‚96.96357250213623 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 7194.09375 disk.sda.in‚0$ disk./.usageGB‚589.9268074035645% disk.nvme0n1.out‚10734.3896484375 disk.rbd1.in‚0 disk.sda.out‚ 115.234375 disk.sda1.in‚0) disk./.usagePercent‚39.59048302641375" disk.dm-0.out‚10734.3896484375 cpu‚26.06523430670129Ü©ÇZ2³- _wandbruntime‚5219  _runtime‚5219vÒ$ Æ´À _step‚490 & lr-AdamW/pg1‚0.0004724904347112885 & lr-AdamW/pg2‚0.0004724904347112885 trainer/global_step‚5999 % _timestamp‚1.7843679413387198e+09 _runtime‚ 5219.84464022³E¾ß˵Š& lr-AdamW/pg1‚0.0004724904347112885 & lr-AdamW/pg2‚0.0004724904347112885 trainer/global_step‚5999 % _timestamp‚1.7843679413387198e+09 _runtime‚ 5219.84464022 _step‚490ê@0l…2¶- _wandbruntime‚5219  _runtime‚5219PÍ­Ïç'·áO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23402506 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657437696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910495951880192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207317587788800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103010746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10511158 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912143585818266 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912143228229016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55515583 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250939392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916828250939392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910495934840832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000767 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55525560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47485184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6322645222808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23402248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚910495951880192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6398389243904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55525560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55515583 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12891090 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905820583006208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916828609317888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6332316098560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916828609317888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6332316098560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200919299442688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207317587788800 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚269 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103010746 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10511158 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚912143585818266 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚912143228229016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1438 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 23842521088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55515583 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚916828250939392 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚916828250939392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910495934840832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 24406654976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10511161 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000767 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55525560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47485184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6322645222808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23402248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚910495951880192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47485186 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚480 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6398389243904 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 564133888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55525560 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55515583 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12891090 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905820583006208 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚916828609317888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 554758144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 23536056591 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚6bJ¦  6332316098560 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚916828609317888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6332316098560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200919299442688 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 357589250 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 564133888 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6332657437696 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1438 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9977 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚910495951880192 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916976504595968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚905967093678080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6323656276530 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47492818 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200952728201216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47492816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12893097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6323996826532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55534626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55524635 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚910643192909824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912291107543972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚5999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916976862981632 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23406257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23406000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6399347027456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905967110717440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976m¢Ïö&½ðM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 % _timestamp‚1.7843679419706476e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚912290749954610 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 564133888 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910643175870464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12893351 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6333328725504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207352149118464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6399420917248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1438 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23406257 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23406000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6399347027456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 153393664 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 341346304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905967110717440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1169 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910643175870464 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6333670071808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6333670071808 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103017451 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6333328725504 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55524635 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12101 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10512903 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103017451 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚916976504595968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚905967093678080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6323656276530 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47492818 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200952728201216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1438 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47492816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12893097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6323996826532 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55534626 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55524635 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 358385664 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚910643192909824 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚912291107543972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 trainer/global_step‚5999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚916976862981632 P&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7452168464660645 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6118022203445435 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37732499837875366 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04252363368868828 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 _runtime‚5238.458171399 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2624202072620392 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3882123827934265 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 .particle/accuracy_3_of_3‚0.999498724937439 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20009422302246094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7692170143127441 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917799830436707 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8120559453964233 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.994635283946991 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5788174867630005 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22957685589790344 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.60254967212677 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0560959167778492 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6203820705413818 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚5999 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6082745790481567 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27257388830184937 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9240250587463379 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6092396974563599 *jet/accuracy_1_of_2‚0.8064939379692078 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_4‚0.9968253374099731 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833134651184 epoch‚14 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9969019293785095 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.81390780210495 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7818216681480408 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11541628837585449 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6489701867103577 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1683940887451172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚.çþ)0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29140543937683105 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2519178092479706 +particle/specificity‚0.9080585241317749 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9957922101020813 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6943236589431763 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31026947498321533 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6115376949310303 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2735140323638916 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10881094634532928 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2761118710041046 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7692170143127441 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.865912675857544 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9957922101020813 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.6352898478507996 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22062213718891144 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11541628837585449 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9932103157043457 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2657102644443512 *jet/accuracy_1_of_3‚0.9305977821350098 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148655891418 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6682651042938232 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5836448073387146 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2638248801231384 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29357144236564636 particle/accuracy_1_of_3‚1 )loss/val_total_loss‚8.799350905849103 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6615787744522095 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.296875 jet/accuracy_5_of_5‚0 _step‚494 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6986255049705505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942231178283691 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6971918344497681 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.34311333298683167 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7003361582756042 % _timestamp‚1.7843679599515276e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17152589559555054 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5349999666213989 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8638149499893188 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930317997932434 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8069539070129395 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.291225790977478 /particle/accuracy_4_of_4‚0.9256236553192139 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29027289152145386 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961599707603455 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6484375 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8923726081848145 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868128895759583 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34549766778945923 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690272092819214 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953330159187317 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46079757809638977 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28024187684059143 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3650068938732147 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 +particle/sensitivity‚0.6220504641532898 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6532694101333618 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30065521597862244 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5444373488426208 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.726236879825592 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6983784437179565 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972735643386841 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6489555835723877 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2112659066915512 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6413005590438843 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29140543937683105 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8141345977783203 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09505146741867065 *jet/accuracy_2_of_3‚0.5477949976921082 *jet/accuracy_1_of_4‚0.9634120464324951 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 +jet/accuracy_3_of_4‚0.09423591941595078 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28288501501083374 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895392060279846 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6516720652580261 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1330859512090683 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48428672552108765 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1423683762550354 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.54109126329422 *jet/accuracy_2_of_4‚0.7832217216491699 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22337132692337036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6128638386726379 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7130573987960815 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3011549115180969 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3203608989715576 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4507681131362915 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6938518285751343 (particle/accuracy‚0.8466370701789856 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.613250195980072 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36889633536338806 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6105629205703735 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_2‚0.1485716998577118 +jet/accuracy_3_of_3‚0.07184718549251556 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05427949130535126 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7298399806022644 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918822050094604 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3007458448410034 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6969753503799438 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3448614776134491 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9858922362327576 7validation_average_jet_accuracy‚0.38328951597213745 jet/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6176409125328064 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7536135911941528 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769IQÈqµ=ů{ 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.22337132692337036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6128638386726379 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7130573987960815 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3011549115180969 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3203608989715576 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4507681131362915 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6938518285751343 (particle/accuracy‚0.8466370701789856 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.613250195980072 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36889633536338806 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6105629205703735 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_2_of_2‚0.1485716998577118 +jet/accuracy_3_of_3‚0.07184718549251556 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05427949130535126 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7298399806022644 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9918822050094604 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3007458448410034 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6969753503799438 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3448614776134491 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9858922362327576 7validation_average_jet_accuracy‚0.38328951597213745 jet/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6176409125328064 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7536135911941528 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1325971782207489 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1411537379026413 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45482078194618225 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5891159772872925 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9237552881240845 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6973273158073425 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6112527251243591 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21208246052265167 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7317249178886414 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6128638386726379 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6345211267471313 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.048976149410009384 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9906299114227295 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7452168464660645 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6118022203445435 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37732499837875366 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04252363368868828 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_4_of_5‚0 particle/accuracy_2_of_5‚1 _runtime‚5238.458171399 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2624202072620392 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974562525749207 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3882123827934265 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 .particle/accuracy_3_of_3‚0.999498724937439 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8055555820465088 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20009422302246094 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7692170143127441 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9917799830436707 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8120559453964233 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.994635283946991 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5788174867630005 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22957685589790344 7"Purity/1FRt0SRqqt1FBt/event_purity‚0.60254967212677 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0560959167778492 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6203820705413818 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚5999 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960317015647888 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6082745790481567 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27257388830184937 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9240250587463379 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6092396974563599 *jet/accuracy_1_of_2‚0.8064939379692078 particle/accuracy_1_of_2‚1 /particle/accuracy_3_of_4‚0.9968253374099731 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3645833134651184 epoch‚14 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9969019293785095 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.81390780210495 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7818216681480408 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.11541628837585449 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6489701867103577 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1683940887451172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.29140543937683105 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2519178092479706 +particle/specificity‚0.9080585241317749 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9957922101020813 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6943236589431763 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.31026947498321533 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6115376949310303 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2735140323638916 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.10881094634532928 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2761118710041046 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7692170143127441 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.865912675857544 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9957922101020813 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.6352898478507996 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.22062213718891144 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11541628837585449 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9932103157043457 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2657102644443512 *jet/accuracy_1_of_3‚0.9305977821350098 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148655891418 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6682651042938232 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.45601850748062134 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5836448073387146 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2638248801231384 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29357144236564636 particle/accuracy_1_of_3‚1 )loss/val_total_loss‚8.799350905849103 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6615787744522095 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.296875 jet/accuracy_5_of_5‚0 _step‚494 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6986255049705505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942231178283691 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6971918344497681 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.34311333298683167 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7003361582756042 % _timestamp‚1.7843679599515276e+09 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17152589559555054 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5349999666213989 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8638149499893188 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930317997932434 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8069539070129395 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 6 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.291225790977478 /particle/accuracy_4_of_4‚0.9256236553192139 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29027289152145386 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9961599707603455 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6484375 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8923726081848145 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9868128895759583 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34549766778945923 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6690272092819214 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953330159187317 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9812596440315247 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.46079757809638977 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.28024187684059143 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3650068938732147 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 +particle/sensitivity‚0.6220504641532898 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6532694101333618 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30065521597862244 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5444373488426208 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.726236879825592 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6983784437179565 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8055555820465088 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972735643386841 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6489555835723877 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2112659066915512 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6413005590438843 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.29140543937683105 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8141345977783203 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.09505146741867065 *jet/accuracy_2_of_3‚0.5477949976921082 *jet/accuracy_1_of_4‚0.9634120464324951 particle/accuracy_3_of_5‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 +jet/accuracy_3_of_4‚0.09423591941595078 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.28288501501083374 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9895392060279846 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6516720652580261 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1330859512090683 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48428672552108765 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1423683762550354 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.54109126329422 *jet/accuracy_2_of_4‚0.7832217216491699làœÆ‚(:‘ ã–íÒ£®µî gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñsÆÇ‚(:» ã–íÒÜŸøð disk.sda.out‚ 118.3828125 disk.sda3.out‚ 118.3828125 disk.drbd2165.out‚0 disk.rbd0.in‚0 network.recv‚ 1.403207e+06 disk.rbd1.in‚0# disk.nvme0c0n1.in‚ 7308.23046875# proc.memory.rssMB‚ 2809.67578125* disk./.usagePercent‚39.590611286188405$ disk./.usageGB‚589.9287185668945 disk.rbd1.out‚13.4375 disk.dm-1.in‚0 disk.sda2.in‚0# disk.nvme0n1.out‚11264.95703125 proc.cpu.threads‚17$ memory_percent‚95.46129703521729 disk.rbd0.out‚0 disk.dm-0.out‚11264.95703125! disk.nvme0n1.in‚ 7308.23046875 disk.dm-0.in‚ 7308.23046875 network.sent‚ 9.687119e+06% disk.nvme0c0n1.out‚11264.95703125 disk.sda.in‚0 disk.dm-1.out‚ 118.3828125 disk.drbd2165.in‚0* proc.memory.percent‚17.148900032043457 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 743.62109375 cpu‚26.07798132167108×ò"c‘È‚(:† ò–íÒ•»‘í gpu.0.gpu‚28.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UîL{ÆÉ‚(:» ò–íÒåÃÔð$ disk./.usageGB‚589.9249572753906 disk.sda1.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 118.7890625" disk.dm-0.out‚11269.6416015625# proc.memory.rssMB‚ 2809.87109375* proc.memory.percent‚17.150092124938965 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0! disk.nvme0n1.in‚ 7308.23046875 network.sent‚ 9.731513e+06 disk.sda.out‚ 118.7890625 disk.sda1.in‚0% disk.nvme0n1.out‚11269.6416015625 cpu‚26.078708391464154 network.recv‚ 1.407505e+06 disk.sda3.out‚ 118.7890625 disk.rbd0.in‚0 disk.dm-0.in‚ 7308.23046875' disk.nvme0c0n1.out‚11269.6416015625 disk.drbd2165.in‚0$ memory_percent‚95.46337127685547% proc.memory.availableMB‚ 743.28125 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 7308.23046875) disk./.usagePercent‚39.59035886275964 disk.sda2.out‚0 disk.rbd1.out‚13.5Áùš,žÊ‚(:“ —íÒ…ÅÚí gpu.0.gpu‚48.0 gpu.0.memory‚13+ gpu.0.memoryAllocated‚29.04792785644531* gpu.0.memoryAllocatedBytes‚ 24951980032 gpu.0.temp‚38.0 gpu.0.powerWatts‚274.502' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.900400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 O]ÃË‚(:¸ —íÒ¾ð disk.sda3.out‚ 119.09375' disk.nvme0c0n1.out‚11275.4111328125 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0 network.sent‚ 9.737053e+06 network.recv‚ 1.41091e+06 disk.sda.out‚ 119.09375$ memory_percent‚95.46353816986084 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚589.9249038696289 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚11275.4111328125* disk./.usagePercent‚39.590355278654165 disk.dm-1.out‚ 119.09375 disk.rbd0.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 7308.23046875 disk.dm-0.in‚ 7308.23046875" disk.dm-0.out‚11275.4111328125( proc.memory.availableMB‚ 743.25390625* proc.memory.percent‚17.150092124938965 cpu‚26.079433879433402 disk.rbd1.out‚13.5 disk.sda.in‚0! disk.nvme0n1.in‚ 7308.23046875# proc.memory.rssMB‚ 2809.87109375 disk.sda3.in‚0és}d2Ì- _wandbruntime‚5281  _runtime‚5281’oß@ÇÍÁ % _timestamp‚1.7843680029568198e+09 _runtime‚ 5281.4627805 _step‚495 ' lr-AdamW/pg1‚0.00047066143852919027 ' lr-AdamW/pg2‚0.00047066143852919027 trainer/global_step‚6049‰e[ÌÎÆ ' lr-AdamW/pg2‚0.00047066143852919027 trainer/global_step‚6049 % _timestamp‚1.7843680029568198e+09 _runtime‚ 5281.4627805 _step‚495 ' lr-AdamW/pg1‚0.00047066143852919027ïä.2Ï- _wandbruntime‚5281  _runtime‚5281#íоáÐâO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13024928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚921445761248838 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47987450 trainer/global_step‚6049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401086147584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6400744808448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚926188297781248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚1—J™H"2154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56050339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6467578221568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919787570012160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56040362 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926188297781248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104027812 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209074309976064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10600931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13025180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926188656159744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104037791 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47987452 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚919787570012160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56050339 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23626111 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚921446118838088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚00Ÿe,í'ÑçO KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1169 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 23702949376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚919787570012160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56050339 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23626111 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚921446118838088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13024928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 24406654976 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 358378496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚921445761248838 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 23025176576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 24406654976 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 23842521088 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1169 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47987450 trainer/global_step‚6049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401086147584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6400744808448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚926188297781248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56050339 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚77 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6467578221568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚269 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 358378496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚919787570012160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 23190607872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 612591616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56040362 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12101 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 153393664 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1438 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926188297781248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1169 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104027812 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 341339136 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1508 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209074309976064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 706451968 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 553967983 _step‚496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10600931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13025180 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚18612224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 564133888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 23702949376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926188656159744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 100897792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚269 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104037791 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1438 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47987452ð0·þ2Ò- _wandbruntime‚5282  _runtime‚5282#"’Áñ&ÓëM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.784368003591397e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402098781696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13027270 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚926336909823488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5282.097681081 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6392333038674 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6391992488672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47995082 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23629680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 358385664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9991 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10602673 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209108657188352 trainer/global_step‚6049 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47995082 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 23842521088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1438 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47995084 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 357589362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13027007 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6468597573120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚269 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚480 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 23842521088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1169 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 564133888 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚926336909823488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚921593640563794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 23536056591 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56049414 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚926336551437824 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104054489 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚202640059615232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1438 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 23702949376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 612591616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚919934811041792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚921593282974432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56049414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224%uÜ‘ÆÔðM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚919934811041792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 23190607872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚921593282974432 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56049414 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 107182080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚18612224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 $ _timestamp‚1.784368003591397e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚77 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6402098781696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13027270 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚269 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 564133888 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚926336909823488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 340550002 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 553967983 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 24406654976 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 23842521088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚269 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5282.097681081 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6392333038674 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6391992488672 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 24406654976 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47995082 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9991 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 554758144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23629680 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 706451968 B